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Executive  Summary 


The  Balanced  Budget  Act  (BBA)  of  1997  significantly  increased  the  number  and  range  of  health 
plan  options  potentially  available  to  Medicare  beneficiaries.  To  inform  Medicare  beneficiaries  about 
these  choices,  the  law  provided  for  specific  information  dissemination  activities  that  the  Centers  for 
Medicare  &  Medicaid  Services  (CMS)  implemented  with  the  National  Medicare  Education  Program 
(NMEP).  The  NMEP  is  the  largest  and  most  comprehensive  information  program  ever  undertaken  by  the 
Medicare  program.  One  important  feature  of  the  NMEP  is  the  Medicare  &  You  handbook  which  was 
first  nationally  distributed  in  1999. 

A  goal  of  RTFs  project,  Analysis  of  Medicare  Beneficiary  Knowledge  Data  Using  the  Medicare 
Current  Beneficiary  Survey  (MCBS):  Phase  Three,  is  to  evaluate  the  impact  of  the  NMEP  on 
beneficiaries'  knowledge  of  the  Medicare  program.  As  a  part  of  this  project,  RTI  is  comparing 
knowledge  before  and  after  the  first  national  distribution  of  the  Medicare  &  You  handbook  using  data 
from  the  1999  and  2000  administrations  of  the  Medicare  Current  Beneficiary  Survey  (MCBS).  To  assess 
beneficiary  knowledge,  we  developed  three  potential  knowledge  measures  using  data  from  the 
Beneficiary  Knowledge  (BK  rounds  23  &  26)  and  Beneficiary  Needs  (BN  rounds  24  &  27)  supplemental 
rounds  of  the  CY  1998  and  1999  Medicare  Current  Beneficiary  Survey  (MCBS)  Access  to  Care  files. 
The  first  measure,  the  perceived  knowledge  index,  includes  five  questions  that  ask  beneficiaries  to 
subjectively  rate  how  much  they  know  about  a  particular  topic  related  to  Medicare.  The  other  two 
measures  are  a  three-item  and  a  seven-item  quiz  that  require  participants  to  respond  to  a  series  of 
true/false  questions. 

We  evaluated  the  psychometric  properties  of  the  knowledge  measures  by  conducting  item-level 
analyses  and  scale-level  descriptive  statistics,  assessing  internal  consistency  reliability,  and  establishing 
construct  validity.  The  item-level  analyses  of  the  perceived  knowledge  items  suggest  that  these  items 
appropriately  do  not  exhibit  floor  or  ceiling  effects.  Comparisons  of  the  3-item  and  7-item  quiz  scores 
according  to  beneficiaries'  educational  achievement  suggest  that  the  difficulty  levels  of  both  of  these 
quizzes  are  equivalent  to  a  high  school  level.  The  reliability  analyses  indicate  that  both  the  perceived 
knowledge  index  and  the  7-item  quiz  reached  acceptable  levels  of  internal  consistency  reliability  while 
the  3-item  quiz  did  not  meet  the  criterion  for  acceptable  reliability. 

We  used  three  different  approaches  to  assess  construct  validity  of  the  knowledge  indices.  First, 
we  evaluated  the  relationship  between  the  knowledge  indices  and  a  global  knowledge  item  that  asks 
beneficiaries  to  rate  how  much  they  know  about  the  Medicare  program.  A  strong  relationship  between  a 
knowledge  index  and  the  global  knowledge  item  would  provide  support  for  the  construct  validity  of  the 
index.  Second,  we  compared  the  mean  knowledge  index  scores  for  groups  previously  shown  to  differ  in 
knowledge.  Finally,  we  compared  the  mean  knowledge  scores  for  groups  that  we  hypothesized  would 
have  different  levels  of  knowledge. 
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Based  solely  on  the  quantitative  results,  the  perceived  knowledge  index  seems  to  have  the  best 
psychometric  properties.  This  index  had  the  highest  Cronbach's  alpha,  demonstrated  a  strong,  monotonic 
relationship  with  the  global  knowledge  question,  and  showed  significant  differences  in  index  scores 
between  respondents  who  were  expected  to  have  different  knowledge  levels.  However,  other  criteria, 
such  as  the  content  of  the  items,  must  be  considered  when  selecting  the  most  appropriate  knowledge 
measure.  The  perceived  knowledge  index  relies  on  beneficiaries  to  be  the  sole  judge  of  their  knowledge. 
Individuals'  subjective  ratings  may  be  influenced  by  factors  other  than  their  knowledge,  such  as 
confidence  in  decision-making  or  satisfaction  with  information  received.  Therefore,  the  perceived 
knowledge  index  may  not  provide  the  most  accurate  assessment  of  actual  knowledge  level.  A  more 
precise  measure  of  knowledge  would  require  respondents  to  demonstrate  their  knowledge,  such  as  by 
correctly  answering  the  true/false  questions  included  on  the  three-item  and  seven-item  quizzes. 

On  the  basis  of  both  content  considerations  and  the  psychometric  analysis  results,  the  seven-item 
quiz  appears  to  be  the  most  useful  measure  of  beneficiary  knowledge  overall.  As  mentioned  above,  the 
quiz  requires  beneficiaries  to  demonstrate  their  knowledge  rather  than  merely  stating  that  they  know 
everything  they  need  to  know.  In  addition,  the  quiz  had  good  variability  in  scores,  reached  an  acceptable 
level  of  internal  consistency  reliability,  and  performed  well  on  the  validity  analyses. 

Possible  modifications  to  the  seven-item  quiz  depend  on  the  intended  use  of  the  measure.  If  the 
goal  of  the  quiz  is  to  obtain  the  most  precise  estimate  of  beneficiaries'  knowledge  possible,  then  the  quiz 
should  contain  questions  that  cover  all  difficulty  levels.  The  grade -level  analyses  suggest  that  the  items 
on  the  seven-item  quiz  cover  all  education  levels  except  for  no  formal  education.  Very  few  respondents 
report  having  no  formal  education  and  therefore  it  may  not  be  necessary  to  add  questions  specifically 
targeted  at  this  group.  However,  the  quiz  contains  only  one  item  at  the  high  school  graduate  level. 
Because  more  respondents  report  being  a  high  school  graduate  than  any  other  educational  level,  it  may  be 
helpful  to  add  more  items  targeted  at  this  group. 

In  contrast,  if  the  purpose  of  the  quiz  is  to  determine  whether  Medicare  beneficiaries  reach  a 
certain  proficiency  level,  then  the  primary  emphasis  should  be  on  the  content  of  the  quiz  items.  The 
items  should  cover  the  entire  range  of  information  that  beneficiaries  need  to  know  in  order  to  make 
informed  decisions.  Less  emphasis  should  be  placed  on  the  difficulty  of  the  items.  In  this  case,  a 
limitation  of  the  seven-item  quiz  is  that  four  of  its  seven  questions  deal  with  managed  care  plans  which 
restricts  the  range  of  knowledge  that  the  quiz  can  measure.  Other  questions  could  be  added  to  improve 
the  comprehensiveness  of  the  quiz.  In  fact,  as  part  of  RTFs  Questionnaire  Development  and  Cognitive 
Testing  Using  Item  Response  Theory  project,  several  new  knowledge  questions  were  developed  that 
address  a  variety  of  issues,  including  beneficiary  rights  and  health  plan  decision-making  (Uhrig  et  al., 
2001),  and  could  be  used  to  expand  the  seven-item  quiz. 
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1.0    Introduction 


The  Balanced  Budget  Act  (BBA)  of  1997  significantly  increased  the  number  and  range  of  health 
plan  options  potentially  available  to  Medicare  beneficiaries.  To  inform  Medicare  beneficiaries  about 
these  choices,  the  law  provided  for  specific  information  dissemination  activities  that  the  Centers  for 
Medicare  &  Medicaid  Services  implemented  with  the  National  Medicare  Education  Program  (NMEP). 
The  NMEP  is  the  largest  and  most  comprehensive  information  program  ever  undertaken  by  the  Medicare 
program.  Some  important  features  of  the  NMEP  include  the  Medicare  &  You  handbook,  the  Medicare 
website  (www.medicare.gov),  and  the  helpline  (1-800-MEDICARE). 

A  goal  of  RTF  s  project,  Analysis  of  Medicare  Beneficiary  Knowledge  Data  Using  the  Medicare 
Current  Beneficiary  Survey  (MCBS):  Phase  Three,  is  to  evaluate  the  impact  of  the  NMEP  on 
beneficiaries'  knowledge  of  the  Medicare  program  and  available  health  plan  options.  The  Medicare 
Current  Beneficiary  Survey  (MCBS)  provides  a  useful  source  of  data  with  which  to  assess  levels  of 
Medicare  beneficiary  program  knowledge.  In  the  MCBS,  a  large  national  probability  sample  of  14,000 
or  more  Medicare  beneficiaries  are  interviewed  in  to  a  rotating  panel  design  every  four  months  for  up  to 
four  years.  Each  year,  approximately  one  quarter  of  the  sample  is  rotated  out  of  the  survey  and  replaced 
with  new  sample  members.  Therefore,  25  percent  of  each  annual  MCBS  data  set  represents  a  cross 
section  of  the  Medicare  population  enrolled  in  the  program  continuously  since  January  1st  of  that  year 
and  75  percent  represents  a  longitudinal  beneficiary  panel. 

As  a  part  of  the  NMEP,  the  Medicare  &  You  (2000)  handbook  was  first  distributed  nationally  in 
the  fall  of  1999.  To  evaluate  the  impact  of  the  handbook,  the  Phase  3  project  assesses  changes  in 
beneficiary  knowledge  from  immediately  before  to  immediately  after  national  distribution  of  the 
handbook.  A  first  step  in  determining  the  level  of  Medicare  program  knowledge  among  Medicare 
beneficiaries  is  to  develop  an  index  to  measure  knowledge  of  the  Medicare  program.  As  a  part  of  Phases 
1  and  2  of  this  project,  knowledge  indices  were  developed  using  data  from  the  CY  1995  to  1998  Access 
to  Care  MCBS  files  (Bann  et  al,  2000).  This  report  expands  on  this  previous  work  by  developing 
knowledge  measures  from  questions  included  in  the  Beneficiary  Knowledge  (BK)  supplemental  rounds 
23  &  26  and  Beneficiary  Needs  (BN)  supplemental  rounds  24  and  27  of  the  1998  and  1999  Access  to 
care  files  MCBS  respectively,  which  were  administered  immediately  before  and  after  national 
distribution  of  the  first  Medicare  &  You  (2000)  handbook1 .  In  addition,  this  report  evaluates  the 
psychometric  properties  of  each  knowledge  measure,  including  internal  consistency  reliability  and 
construct  validity. 


1  The  1998  and  1999  BK  supplements  (Rounds  23  &  26)  were  administered  from  January  to  April  of  1999  and  2000, 
respectively,  while  the  1998  and  1999  BN  (supplemental  rounds  24  &  27)  were  administered  from  May  to  August 
of  1999  and  2000,  respectively. 


2. 0    Development  of  Knowledge  Measures 


We  developed  three  different  knowledge  measures  using  questions  from  the  CY  1998  and  1999 
Access  to  Care  files  and  related  supplemental  rounds  of  the  MCBS.  The  knowledge  measures  and  the 
items  comprising  them  are  listed  in  Table  A-l  in  the  appendix.  The  first  knowledge  measure,  the 
perceived  knowledge  index,  includes  five  items  that  were  administered  during  BN  Rounds  24  and  27  . 
These  questions  ask  beneficiaries  how  much  they  feel  they  know  about  five  topics:  (a)  what  services 
Medicare  covers,  (b)  how  much  they  have  to  pay  for  medical  services  covered  by  Medicare, 
(c)  supplemental  or  Medigap  insurance,  (d)  the  availability  and  benefits  of  Medicare  HMOs,  and 
(e)  choosing  or  finding  a  doctor  or  other  health  care  provider. 

The  other  two  measures  are  quizzes  requiring  beneficiaries  to  determine  whether  statements  are 
true  or  false.  The  three-item  quiz,  which  was  administered  during  BN  Rounds  24  and  27,  includes 
true/false  questions  about  whether  Medicare  covers  colorectal  cancer  screening,  whether  it  pays  for  flu 
shots,  and  whether  supplemental  insurance  is  the  same  as  a  Medicare  managed  care  plan.  The  seven-item 
quiz,  administered  in  BK  Rounds  23  and  26,  includes  questions  about  Medicare  options  and  Medicare 
managed  care  plans.  The  quiz  contained  eight  items  in  19982;  however,  one  of  these  items  was  dropped 
from  the  1999  MCBS.  To  allow  for  comparisons  across  1998  and  1999,  only  the  seven  items  included  in 
both  years  were  used  to  compute  the  quiz  scores  in  this  report. 


2.1      Scoring  Algorithms 

2.1.1    Perceived  Knowledge  Index 

The  perceived  knowledge  index  was  created  by  reverse  scoring  each  of  the  five  response 
categories  across  the  items.  For  example,  in  the  original  coding  of  the  variables,  knowing  "just  about 
everything  you  need  to  know..."  was  coded  as  "1,"  and  knowing  "almost  none  of  what  you  need  to 
know. .."  was  coded  as  "5."  The  former  response  was  recoded  as  "5,"  while  the  latter  was  recoded  as 
"1."  Responses  for  "2"  and  "4"  were  also  switched.  "Don't  know"  and  "Refused"  responses  were 
recoded  as  missing.  Then  the  recoded  responses  for  all  five  questions  were  summed  to  compute  the 
overall  perceived  knowledge  index  score. 

For  example,  if  a  respondent  answered  "3,"  "4,"  "2,"  "1,"  and  "5,"  respectively,  to  the  original 
questions,  the  questions  would  be  reverse  coded  and  the  knowledge  score  would  be  calculated  as  follows: 

3  +  2  +  4  +  5  +  1  =  15. 


The  perceived  knowledge  index  was  referred  to  as  the  "know-all-need-to-know  index"  in  Bann  et  al.  (2000). 
A  psychometric  evaluation  of  the  eight-item  quiz  is  included  in  Bann  et  al.  (2000). 
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Higher  scores  on  this  index  reflect  beneficiaries  reporting  that  they  knew  more  of  what  they  needed  to 
know  about  the  five  different  topics.  Scores  produced  by  this  system  may  range  from  5  to  25,  thus 
providing  greater  variability  in  scores  and  more  power  to  discriminate  among  beneficiaries  than  would  be 
obtained  with  only  true/false  (two  response)  questions. 

We  excluded  some  potential  survey  questions  from  the  index.  One  item  regarding  the  Medicaid 
program  was  not  included  because  it  was  asked  of  Medicaid  recipients  only.  Another  item  that  asks 
whether  respondents  know  what  they  need  to  know  to  stay  healthy  was  excluded  from  the  calculation  of 
the  scale  because  the  question  does  not  address  an  insurance  benefit  or  option  and  therefore  appeared 
conceptually  unrelated  to  the  other  five  items. 

We  calculated  the  perceived  knowledge  index  by  imputing  values  for  missing  data  rather  than 
eliminating  a  respondent's  answers.  The  mean  of  the  remaining  items  was  substituted  for  the  missing 
item  values  (Chapman,  1976).  Imputation  was  used  only  for  respondents  who  answered  at  least  half  of 
the  items  (three  of  the  five  items).  Individuals  missing  responses  to  more  than  half  of  the  items  were 
assigned  a  value  of  missing  for  the  index. 


2.1.1    Three-Item  Quiz 

Correct  responses  to  each  of  the  three  quiz  questions  were  coded  as  "1,"  while  incorrect  or 
"don't  know"  responses  were  coded  as  "0."  One  potential  quiz  item  concerning  Medicare  coverage  of 
mammograms  was  excluded  from  the  overall  score  calculation  because  it  was  asked  only  of  women.  The 
recoded  responses  were  then  summed  to  create  the  quiz  scores.  For  example,  a  respondent  who  had  only 
one  correct  response  and  two  incorrect  responses  received  a  score  of  "1,"  as  shown: 

0+  1  +0  =  1. 

An  advantage  of  this  scoring  method  is  that  it  yields  scores  with  meaningful  interpretations,  specifically, 
the  number  of  questions  to  which  the  respondent  knew  the  correct  answer.  Scores  range  from  "0"  to  "3," 
with  higher  scores  indicating  greater  knowledge  of  the  Medicare  program. 

For  both  this  quiz  and  the  seven-item  quiz  described  below,  respondents  who  refused  or  were 
missing  data  on  all  of  the  quiz  items  were  coded  as  missing  for  the  overall  quiz  score.  In  this  sample,  all 
of  the  remaining  respondents  provided  answers  to  all  of  the  quiz  questions  or  gave  a  response  of  "don't 
know."  As  mentioned  earlier,  "don't  know"  responses  were  coded  as  incorrect.  Therefore,  in  contrast  to 
the  perceived  knowledge  index,  we  did  not  need  to  impute  missing  values  in  order  to  calculate  scores  for 
the  three-item  quiz  or  the  seven-item  quiz. 
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2.1.3   Seven-Item  Quiz 

The  score  for  the  seven-item  quiz  was  created  using  a  scoring  method  similar  to  that  of  the  three- 
item  quiz.  Correct  responses  were  given  a  score  of  "1,"  and  incorrect  and  "don't  know"  responses  were 
coded  as  "0."  The  recoded  items  were  then  summed  to  create  an  overall  quiz  score.  For  example,  the 
score  for  a  respondent  who  answered  the  first  four  items  correctly  and  the  last  three  incorrectly  would  be 
calculated  as  follows: 

1  +  1+1+1+0  +  0  +  0  =  4. 

This  respondent  would  receive  a  quiz  score  of  "4,"  indicating  that  four  of  the  seven  questions  were 
answered  correctly.  With  this  scoring  system,  quiz  scores  can  range  from  "0"  to  "7,"  with  higher  scores 
indicating  greater  levels  of  knowledge. 
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3. 0    Psychometric  Analysis  Methods 


To  evaluate  the  psychometric  properties  of  the  knowledge  measures,  we  conducted  item-level 
and  scale-level  analyses.  The  item-level  analyses  examined  the  psychometric  properties  of  each 
individual  item  included  in  the  scale,  while  the  scale-level  analyses  evaluated  the  scale  as  a  whole. 

3.1  Item-Level  Analyses 

For  all  three  knowledge  indices,  we  computed  the  correlations  between  each  item  on  the  index 
and  the  total  index  score.  Item-total  score  correlations  provide  information  about  the  contribution  of  the 
item  to  the  reliability  of  the  scale.  Ideally,  items  should  have  item-total  correlations  of  at  least  0.30. 
Because  the  item  itself  is  included  in  the  calculation  of  the  index  score,  the  item-total  score  correlations 
may  be  inflated.  To  avoid  possible  inflation,  the  item-total  score  correlations  in  this  report  were 
corrected  for  overlap  (Howard  and  Forehand,  1962). 

For  the  perceived  knowledge  items,  we  examined  the  distribution  of  responses  for  possible  floor 
or  ceiling  effects  (i.e.,  scale  compression).  Ideally,  the  item  should  have  variability  in  responses, 
indicating  that  participants  are  utilizing  all  five  response  options.  For  the  true/false  quiz  questions,  we 
examined  the  percentage  of  correct  responses  which  provides  an  indicator  of  the  difficulty  of  the 
questions.  In  addition,  we  utilized  information  on  beneficiaries'  highest  school  grade  completed  to  try  to 
match  the  difficulty  level  of  an  item  to  a  grade  level.  If  possible,  a  knowledge  index  should  contain  items 
with  a  wide  range  of  difficulty  levels  to  enable  it  to  discriminate  among  respondents  with  a  variety  of 
knowledge  levels. 

3.2  Scale-Level  Analyses 

This  section  describes  the  analyses  used  to  assess  the  psychometric  properties  of  the  knowledge 
indices.  For  each  scale,  three  sets  of  analyses  were  computed:  (1)  descriptive  statistics,  (2)  reliability 
analyses,  and  (3)  validity  analyses. 

3.2.1    Descriptive  Statistics 

For  each  scale,  we  calculated  descriptive  statistics  to  determine  the  most  representative  scale 
scores  and  to  examine  the  distribution  of  scores.  A  lack  of  variability  of  scores  can  compromise  the 
validity  of  scale  scores.    We  report  the  mean,  median,  mode,  and  standard  deviation  for  both  the 
perceived  knowledge  index  and  the  7-item  quiz.  We  report  the  frequency  of  scores  for  the  three-item 
quiz  because  it  has  only  four  possible  total  quiz  scores. 
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3.2.2  Reliability 

The  internal  consistency  reliability  of  the  scales  was  estimated  using  Cronbach's  alpha 
coefficient  (Cronbach,  1951).  Internal  consistency  measures  the  degree  to  which  items  on  a  scale  are 
related  to  each  other  and  therefore  appear  to  be  measuring  the  same  construct.  One  common  rule  of 
thumb  is  to  require  alpha  coefficients  to  be  0.70  and  above  in  order  for  the  index  score  to  be  considered 
reliable  for  use  in  group-level  statistical  analyses  (Guilford,  1956;  Nunally,  1978).  The  coefficient 
alphas  for  each  index  were  also  calculated  separately  for  several  subgroups  defined  by  insurance  and 
service  utilization  variables.  These  analyses  helped  determine  whether  the  reliability  of  the  index 
remained  consistent  across  different  groups. 

Test-retest  reliability  was  not  assessed  because  this  type  of  reliability  is  used  to  measure  the 
stability  of  a  scale  over  time  and  is  usually  assessed  over  a  short  period.  The  time  between  the 
administration  of  each  wave  of  the  knowledge  supplement  MCBS  is  relatively  long  (i.e.,  one  year)  during 
which  time  several  factors  (e.g.,  experience  with  the  program,  use  of  services)  could  affect  a  respondent's 
level  of  knowledge.  Calculating  the  test-retest  reliability  using  assessments  administered  so  far  apart 
would  greatly  underestimate  the  reliability  of  the  scales.  Therefore,  test-retest  reliability  is  not  an 
appropriate  type  of  reliability  assessment  for  these  knowledge  measures. 

3.2.3  Validity 

Several  different  approaches  may  be  used  to  establish  the  validity  of  a  new  scale.  Ideally,  the 
new  scale  would  be  shown  to  be  highly  related  to  a  well-established  and  validated  scale  measuring  the 
same  construct,  often  called  a  "gold  standard."  Unfortunately,  a  gold  standard  for  measuring  Medicare 
beneficiary  knowledge  does  not  exist.  Therefore,  we  assessed  validity  using  three  alternative 
approaches.  These  methods  may  not  individually  provide  evidence  as  strong  as  that  of  a  comparison  with 
a  gold  standard,  but  together  they  may  provide  at  least  preliminary  evidence  to  support  the  validity  of  the 
knowledge  indices. 

For  the  first  set  of  validity  analyses,  we  examined  the  relationship  between  the  knowledge 
indices  and  another  measure  of  the  same  construct.  If  the  indices  are  highly  related  to  the  other 
knowledge  measure,  they  will  have  demonstrated  evidence  of  construct  validity.  The  MCBS  includes  a 
knowledge  question  that  asks  respondents  how  much  they  feel  they  know  about  the  Medicare  program; 
the  question  is  followed  by  a  5-point  rating  scale  ranging  from  "almost  none  of  what  you  need  to  know" 
to  "just  about  everything  you  need  to  know."  Bann  and  colleagues  (2000)  refer  to  this  question  as  the 
"global  know-all-need-to-know  (KANTK)  question."  This  question  obtains  the  respondents'  own 
perceptions  of  their  Medicare-related  knowledge  and  would  be  expected  to  be  related  to  the  level  of 
knowledge  indicated  by  their  scores  on  the  knowledge  indices.  A  strong  monotonic  relationship  between 
this  question  and  the  knowledge  indices  would  support  the  validity  of  the  indices. 

Next  we  conducted  analyses  to  determine  if  the  knowledge  scale  scores  discriminated  among 
groups  of  Medicare  beneficiaries  who  have  previously  been  shown  to  differ  in  their  knowledge  of 
Medicare.  This  approach  is  sometimes  referred  to  as  known-groups  comparisons.  We  based  our 
expectations  of  differing  levels  of  knowledge  among  beneficiaries  on  previous  research.  For  example, 
factors  related  to  socioeconomic  status  are  often  predictive  of  levels  of  insurance  knowledge.  Several 
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studies  report  that  respondents  with  more  education  have  higher  levels  of  insurance  knowledge  (Lambert, 
1980;  Marquis,  1983;  McCall,  Rice,  and  Sangl,  1986;  Hibbard  et  al.,  1998;  McCormack  et  al.,  2002). 
Higher  knowledge  levels  have  also  been  associated  with  higher  incomes  (Lambert,  1980;  Marquis,  1983; 
McCall,  Rice,  and  Sangl,  1986;  Hibbard  et  al.,  1998)  and  having  a  supplemental  insurance  plan 
(Cafferata,  1984).  Other  researchers  have  found  that,  among  older  adults,  those  who  are  younger  have 
more  insurance-related  knowledge  (Lambert,  1980;  Cafferata,  1984). 

Based  on  this  research,  we  expected  that  the  following  groups  of  beneficiaries  would  have  higher 
levels  of  knowledge  about  the  Medicare  program:  (1)  beneficiaries  with  more  education,  (2)  beneficiaries 
with  higher  incomes,  and  (3)  beneficiaries  with  supplemental  insurance.  We  also  expected  that  among 
beneficiaries  who  are  eligible  for  Medicare  because  of  their  age,  those  who  are  younger  would  have  more 
knowledge.  In  this  report,  evidence  for  construct  validity  of  a  particular  scale  is  provided  if  the  results  of 
the  analyses  on  that  scale  showed  these  expected  patterns. 

Finally,  we  compared  groups  of  respondents  that  we  hypothesized  would  differ  in  knowledge. 
For  this  report,  we  called  these  exploratory  comparisons.  We  expected  that  beneficiaries  enrolled  in 
managed  care  during  the  past  year  would  have  more  knowledge  than  those  not  enrolled  in  managed  care 
during  the  past  year  because  they  are  required  to  make  more  choices  regarding  their  insurance 
arrangements  than  beneficiaries  using  fee-for-service.  Because  four  of  the  items  on  the  seven-item  quiz 
address  issues  involving  managed  care,  we  expected  the  greatest  effect  to  be  present  for  this  measure. 

In  addition,  we  hypothesized  that  beneficiaries  who  have  more  experience  with  the  Medicare 
program  would  have  higher  levels  of  knowledge.  For  these  validity  analyses,  we  used  level  of  service 
utilization  as  an  approximation  of  experience  with  the  Medicare  program.  Two  types  of  service 
utilization  during  the  past  year  were  included:  (1)  institutional  utilization  and  (2)  Part  B  utilization.  The 
amounts  of  allowable  and  reimbursed  charges  were  also  used  as  indicators  of  experience  with  the 
Medicare  system. 

Some  prior  research  supports  this  hypothesis.  Cafferata  (1984)  found  that  among  a  subsample  of 
older  adults  with  private  insurance,  service  utilization  was  positively  associated  with  knowledge. 
McCormack  and  colleagues  (2002)  found  that  hospitalization  and  number  of  doctor  visits  were  positively 
related  to  beneficiary  knowledge  of  the  Medicare  system.  Results  from  the  national  Medicare  &  You 
evaluation  also  suggest  a  positive  relationship  between  beneficiary  knowledge  and  number  of  doctor 
visits  (McCormack  et  al.,  2001). 

The  following  background  and  experience  variables  were  used  for  both  the  known-groups 
comparisons  and  the  exploratory  comparisons: 

•  income  (under  $25,000  or  $25,000  or  more), 

•  age  (65  to  75  years  old,  or  over  75  years  old), 

•  educational  achievement  (8th  grade  or  less,  more  than  8th  grade  but  no  college,  or  college), 

•  enrollment  in  managed  care  during  the  past  year  (enrolled  or  not  enrolled), 
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•  private  supplemental  insurance  (have  supplemental  insurance  or  do  not  have  supplemental 
insurance), 

•  institutional  utilization  (some  utilization  or  no  utilization), 

•  Part  B  utilization  (some  utilization  or  no  utilization), 

•  total  covered  charges  ($0;  $  1  to  $499;  $500  to  $4,999;  or  $5,000  or  more), 

•  total  reimbursed  dollars  ($0;  $1  to  $499;  $500  to  $4,999;  or  $5,000  or  more), 

•  covered  institutional  charges  ($0;  $1  to  $499;  $500  to  $4,999;  or  $5,000  or  more),  and 

•  Part  B  charges  ($0;  $1  to  $499;  $500  to  $1,499;  or  $1,500  or  more). 

Because  complete  information  on  service  utilization  was  available  only  for  respondents  who  were  not 
enrolled  in  an  HMO,  our  analyses  of  these  variables  included  only  individuals  who  were  not  enrolled  in 
managed  care  during  the  year  before  the  survey  data  were  collected. 
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4.0    Results 


This  section  describes  the  results  of  the  psychometric  analyses  of  the  three  knowledge  indices: 
the  perceived  knowledge  index,  the  three-item  quiz,  and  the  seven-item  quiz.  For  each  index,  both  item- 
level  and  scale-level  analyses  were  computed.  As  a  part  of  the  scale-level  analyses,  the  reliability  and 
construct  validity  of  the  indices  were  evaluated. 

Data  from  the  CY  1998  and  1999  Access  to  Care  Beneficiary  Knowledge  (BK  rounds  23  &  26) 
and  Beneficiary  Needs  (BN  rounds  24  &  27)  supplemental  rounds  of  the  MCBS  were  used  for  the 
psychometric  analyses.  During  1 998,  the  knowledge  questions  were  administered  to  all  respondents, 
however,  in  1 999  the  questions  comprising  the  three-item  quiz  and  the  seven-item  quiz  were  only 
administered  to  respondents  who  were  in  their  first  year  of  participation  in  the  MCBS.  Therefore,  a 
much  smaller  number  of  respondents  was  available  for  the  1999  analyses  of  these  indices. 

Only  participants  who  were  living  in  the  community,  rather  than  an  institution,  were  included  in 
the  psychometric  analyses.  In  addition,  because  this  was  an  elderly  population  whose  members  were 
likely  to  experience  disabilities,  the  use  of  a  proxy  was  sometimes  necessary  to  obtain  information  on  a 
respondent.1  Therefore,  for  completeness,  proxy  information  was  included  in  these  analyses.  However, 
for  each  of  these  comparisons,  data  for  sample  members  and  proxy  respondents  were  analyzed 
separately.  It  was  expected  that  proxy  and  sample  member  participants  would  respond  to  the  knowledge 
indices  differently. 


4.1     Perceived  Knowledge  Index 
4.1.1    Item-Level  Analyses 

Each  item  in  the  perceived  knowledge  index  contains  these  five  response  options:  (1)  "Almost 
none  of  what  you  need  to  know,"  (2)  "A  little  of  what  you  need  to  know,"  (3)  "Some  of  what  you  need  to 
know,"  (4)  "Most  of  what  you  need  to  know,"  and  (5)  "Just  about  everything  you  need  to  know." 
Because  these  items  had  more  than  two  response  options,  we  examined  the  distribution  of  responses  for 
any  possible  floor  or  ceiling  effects.  A  floor  effect  would  be  present  if  respondents  tended  to  select  only 
the  lowest  response  options,  while  a  ceiling  effect  would  occur  if  respondents  selected  only  the  highest 
response  options.  The  presence  of  either  of  these  effects  would  restrict  the  possible  range  of  scores  and 
thereby  limit  the  ability  of  the  index  scores  to  discriminate  among  respondents  with  different  levels  of 
knowledge. 


1   In  both  1998  and  1999,  proxy  interviews  comprised  10%  of  the  interviews  conducted. 
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Table  1  presents  the  response  distributions  of  the  perceived  knowledge  questions  for  the  sample 
members  in  1998  and  1999  while  Table  2  presents  the  response  distributions  for  proxy  respondents. 
Overall,  the  perceived  knowledge  items  demonstrated  good  variability  across  responses,  suggesting  that 
there  are  no  floor  or  ceiling  effects  for  any  of  the  items. 

Table  1.  Response  Distribution  of  the  Perceived  Knowledge  Questions  among 
Sample  Members 


Sample  Members  - 1998 

Question 

Almost  None 

A  little 

Some 

Most 

Just  about  everything 

Services  Medicare  covers 

13% 

17% 

28% 

29% 

13% 

Paying  for  medical  services 

13% 

16% 

24% 

29% 

18% 

Supplemental  insurance 

26% 

16% 

20% 

23% 

15% 

Medicare  HMOs 

41% 

17% 

16% 

15% 

11% 

Choosing  a  doctor 

8% 

11% 

20% 

34% 

28% 

Sample  Members  - 1999 

Question 

Almost  None 

A  little 

Some 

Most 

Just  about  everything 

Services  Medicare  covers 

11% 

16% 

27% 

31% 

16% 

Paying  for  medical  services 

11% 

15% 

24% 

31% 

20% 

Supplemental  insurance 

25% 

15% 

19% 

25% 

16% 

Medicare  HMOs 

41% 

15% 

15% 

15% 

13% 

Choosing  a  doctor 

7% 

9% 

19% 

34% 

30% 

SOURCE:  Centers  for  Medicare  &  Medicaid  Services,  Medicare  Current  Beneficiary  Survey  1998  and  1999  Access 
to  Care  and  Supplemental  Files  BN  24  &  27. 

i 

Table  2.  Response  Distribution  of  the  Perceived  Knowledge  Questions  among 
Proxy  Respondents 


Proxy  Respondents  - 1998 

Question 

Almost  None 

A  little 

Some 

Most 

Just  about  everything 

Services  Medicare  covers 

15% 

18% 

31% 

24% 

12% 

Paying  for  medical  services 

14% 

15% 

24% 

27% 

20% 

Supplemental  insurance 

30% 

19% 

18% 

20% 

13% 

Medicare  HMOs 

42% 

20% 

16% 

13% 

9% 

Choosing  a  doctor 

8% 

12% 

20% 

34% 

27% 

Proxy  Respondents  - 1999 

Question 

Almost  None 

A  little 

Some 

Most 

Just  about  everything 

Services  Medicare  covers 

16% 

15% 

28% 

27% 

14% 

Paying  for  medical  services 

16% 

11% 

22% 

29% 

22% 

Supplemental  insurance 

31% 

15% 

17% 

21% 

16% 

Medicare  HMOs 

45% 

16% 

15% 

14% 

11% 

Choosing  a  doctor 

8% 

10% 

19% 

35% 

28% 

SOURCE:  Centers  for  Medicare  &  Medicaid  Services,  Medicare  Current  Beneficiary  Survey  1998  and  1999  Access 
to  Care  and  Supplemental  Files  BN  24  &  27. 
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Table  3  displays  the  item-total  score  correlations  for  the  perceived  knowledge  items  in  1998  and 
1999.  The  correlations  are  very  similar  across  both  years.  All  of  the  correlations  are  0.5  or  greater, 
suggesting  that  these  items  are  highly  related  and  appear  to  be  measuring  the  same  construct.  The  first 
two  items  (services  Medicare  covers  and  paying  for  medical  services)  are  the  most  highly  related  to  the 
underlying  construct  with  item-total  score  correlations  around  0.7.  The  item  about  Medicare  HMOs  is 
the  least  related  to  the  construct  with  correlations  of  approximately  0.5;  however,  it  still  contributes  to 
the  reliability  of  the  index. 


Table  3.  Item-Total  Score  Correlations  for  the  Perceived  Knowledge  Questions 


Question 

Sample  Member  Interviews 

Proxy  Interviews 

1998 

1999 

1998 

1999 

Services  Medicare  covers 

0.71 

0.71 

0.72 

0.74 

Paying  for  medical  services 

0.69 

0.70 

0.74 

0.72 

Supplemental  insurance 

0.62 

0.63 

0.70 

0.70 

Medicare  HMOs 

0.50 

0.50 

0.54 

0.54 

Choosing  a  doctor 

0.56 

0.56 

0.61 

0.62 

SOURCE:  Centers  for  Medicare  &  Medicaid  Services,  Medicare  Current  Beneficiary  Survey  1998  and 
1999  Access  to  Care  and  Supplemental  Files  BN  24  &  27. 


4.1.2   Scale-Level  Analyses 

This  section  describes  the  scale-level  descriptive  statistics  and  the  reliability  and  validity 
analyses  for  the  perceived  knowledge  index. 

Descriptive  Statistics 

The  descriptive  statistics  for  the  perceived  knowledge  index  are  presented  in  Table  4.  For  both 
sample  member  and  proxy  respondents,  the  mean  scores  on  the  perceived  knowledge  index  are  slightly 
higher  in  1999  than  in  1998.  The  mode  for  sample  members  in  1999  is  higher  than  the  mean  and  median, 
suggesting  that  the  distribution  of  scores  is  skewed  to  the  right.  In  other  words,  the  majority  of  scores  are 
clustered  on  the  higher  end  of  the  scale,  suggesting  that  most  respondents  received  scores  higher  than  15 
(the  middle  possible  score).  The  distribution  of  scores  for  proxy  respondents  in  1998  is  also  slightly 
skewed  to  the  right.  The  distributions  for  proxies  and  sample  members  in  1999  appear  to  be  fairly 
normally  distributed  with  similar  values  for  the  means,  medians,  and  modes. 


Table  4.  Descriptive  Statistics  for  the  Perceived  Knowledge  Index 


Sample  Member  Interviews  (Survey  Year) 

N 

Mean 

S.D. 

Median 

Mode 

1998  (Round  24) 

12,524 

15.2 

5.0 

15 

15 

1999  (Round  27) 

12,606 

15.6 

5.0 

16 

20 

Proxy  Interviews  (Survey  Year) 

N 

Mean 

S.D. 

Median 

Mode 

1998  (Round  24) 

1,330 

14.9 

5.6 

15 

17 

1999  (Round  27) 

1,382 

15.1 

5.3 

15 

15 

Note:  The  possible  range  of  scores  for  the  perceived  knowledge  index  was  from  5  to  25. 
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Reliability 

The  value  of  Cronbach's  alpha  was  0.82  for  sample  members  in  both  1998  and  1999.  Similar 
values  for  coefficient  alpha  were  obtained  for  proxy  respondents  in  1 998  (a  =  0.84)  and  1 999  (a  =  0.85). 
These  values  indicate  that  the  perceived  knowledge  index  demonstrated  strong  internal  consistency 
reliability. 

Coefficient  alphas  of  the  perceived  knowledge  index  were  also  calculated  separately  for  various 
subgroups  classified  according  to  enrollment  in  managed  care,  institutional  and  Part  B  utilization,  and 
amounts  of  allowable  and  reimbursed  charges.  For  details  of  these  results,  please  refer  to  Tables  A-2 
through  A-8  of  the  appendix.  The  coefficient  alphas  of  these  different  groups  were  very  similar.  Among 
sample  member  respondents,  values  of  alpha  ranged  from  0.82  to  0.83  in  1998  and  from  0.81  to  0.84  in 
1999.  Proxy  respondents  had  similar  values  with  alphas  ranging  from  0.81  to  0.87  in  1998  and  from  0.82 
to  0.87  in  1999. 

Validity 

Relationship  with  Global  KANTK  Question 

We  assessed  the  construct  validity  of  the  perceived  knowledge  index  with  three  different 
approaches.  First  we  evaluated  the  relationship  between  the  perceived  knowledge  index  and  a  global 
knowledge  item  that  asks  beneficiaries  how  much  they  feel  they  know  about  the  Medicare  program. 
Analyses  of  variance  (ANOVAs)  indicated  that  perceived  knowledge  index  scores  varied  significantly 
across  levels  of  the  global  KANTK  question  for  sample  members  in  both  1998  (F(4,  12336)  =  762.71, 
p  <  .0001)  and  1999  (F(4,  3617)  =  250.33,  p  <  .0001).  Similar  results  were  found  for  proxy  respondents 
in  1998  (F(4,  1323)  =  67.82,  p  <  .0001)  and  1999  (F(4,  369)  =  17.16,  p  <  .0001).  This  means  the 
perceived  knowledge  index  scores  were  related  to  the  KANTK  scores  other  than  by  chance.  To  help 
interpret  the  results,  we  computed  the  means  and  standard  deviations  of  the  perceived  knowledge  index 
scores  for  each  of  the  five  response  categories  included  in  the  global  KANTK  question.  As  shown  in 
Table  5,  there  is  a  clear,  monotonic  relationship  between  the  perceived  knowledge  index  scores  and  the 
global  KANTK  question  across  both  years  and  both  types  of  respondents.  Individuals  who  rated  their 
knowledge  higher  on  the  global  KANTK  question  had  higher  scores  on  the  perceived  knowledge  index. 
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Table  5.   Means  (and  Standard  Deviations)  of  Perceived  Knowledge  Index 
Scores  by  Response  to  Global  KANTK  Question 


Sample  Member  Interviews 

Survey  Year 

Almost  none 

A  little 

Some 

Most 

Just  about 
everything 

1998  (Round  24) 

11.8(5.0) 

13.4(4.5) 

15.3  (4.3) 

17.5  (4.2) 

18.7(4.7) 

1999  (Round  27) 

11.6(4.8) 

13.9  (4.7) 

15.6(4.4) 

17.8  (4.0) 

18.9  (4.4) 

Proxy  Interviews 

Survey  Year 

Almost  none 

A  little 

Some 

Most 

Just  about 
everything 

1998  (Round  24) 

11.7(5.2) 

13.3(4.6) 

15.0(4.3) 

17.2  (4.3) 

18.4(5.3) 

1999  (Round  27) 

11.6(5.0) 

14.3  (5.3) 

15.2(5.2) 

16.4  (5.2) 

20.0(5.2) 

Group  Comparisons 

For  the  next  set  of  validity  analyses,  we  compared  the  perceived  knowledge  index  scores  of 
beneficiaries  who  have  been  shown  in  previous  research  to  have  different  levels  of  knowledge;  we 
referred  to  this  set  of  analyses  as  known-groups  comparisons.  As  another  set  of  validity  analyses,  we 
compared  beneficiaries  who  we  hypothesized  would  have  different  levels  of  knowledge  which  we 
referred  to  as  exploratory  comparisons.  In  both  sets  of  analyses,  the  index  value  validity  measures  were 
computed  separately  for  sample  member  and  proxy  respondents.  T-tests  were  used  to  compare  the 
perceived  knowledge  index  scores  forsample  members  and  proxy  respondents.  As  hypothesized,  sample 
member  respondents  had  significantly  different  scores  on  the  perceived  knowledge  index  than  proxy 
respondents  both  during  1998  (t(13883)=2.97,  p  =  .003)  and  1999  (t(1660)=3.35,  p  <  .001). 

ANOVAs  and  t-tests  were  used  to  compare  the  perceived  knowledge  index  scores  of  respondents 
according  to  the  various  background  and  experience  variables.  For  the  t-tests,  the  assumption  of  equal 
variance  for  the  two  groups  was  tested  using  the  F'  (folded)  statistic  (Steel  &  Torrie,  1980).  As 
implemented  in  SAS,  if  this  test  was  significant  (i.e.,  the  variances  for  the  two  groups  were  not  equal),  an 
approximate  t  statistic  was  computed,  using  Satterthwaite's  approximation  to  estimate  the  degrees  of 
freedom  (Satterthwaite,  1946). 

The  mean  perceived  knowledge  index  scores  for  each  of  the  groups  are  presented  separately 
according  to  interview  type  in  Tables  A-9  and  A-10.  Asterisks  are  used  to  denote  the  significance  of  the 
relevant  ANOVA  or  t-test.  As  shown  in  Table  A-9,  perceived  knowledge  index  scores  for  sample 
members  in  both  years  differed  significantly  on  all  of  the  variables.  Examining  the  patterns  of  means 
indicates  that  the  results  are  generally  in  the  expected  direction.  Sample  members  with  more  education 
and  higher  incomes  received  higher  knowledge  index  scores.  Also,  higher  scores  were  found  for  sample 
members  between  65  and  75  years  of  age,  with  some  institutional  utilization,  some  Part  B  utilization, 
private  supplemental  insurance,  or  enrollment  in  managed  care.    However,  the  means  of  the  charges 
variables  did  not  always  follow  an  entirely  monotonic  pattern,  possibly  due  to  the  number  and  range  of 
cut-off  points  chosen.  In  some  cases,  the  mean  knowledge  index  scores  for  respondents  with  the  most 
charges  (e.g.,  $5000  or  more)  were  similar  or  smaller  than  the  mean  scores  for  respondents  with  fewer 
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charges  (e.g.,  $500-$4,999).  However,  overall,  there  does  appear  to  be  a  general  pattern  with  individuals 
who  have  no  charges  receiving  lower  knowledge  scores  than  those  with  any  charges. 

Table  A-10  displays  the  results  for  proxy  respondents  in  1998  and  1999.  There  were  significant 
differences  for  all  variables,  except  for  managed  care  in  1998  and  1999  and  institutional  utilization  and 
private  supplemental  insurance  in  1999.  The  patterns  of  means  are  very  similar  to  those  among  sample 
members.  Higher  knowledge  scores  were  found  among  respondents  with  higher  education,  higher 
incomes,  some  Part  B  utilization,  or  any  charges. 


4.2     Three-Item  Quiz 

This  section  outlines  the  results  of  the  item-level  and  scale-level  analyses  of  the  three-item  quiz. 

4.2.1    Item-Level  Analyses 

While  the  perceived  knowledge  questions  ask  respondents  to  rate  their  own  knowledge,  the 
questions  on  the  three-item  quiz  require  respondents  to  demonstrate  their  knowledge  by  determining 
whether  a  statement  is  true  or  false.  Therefore,  participants'  responses  to  these  questions  may  be 
classified  as  correct  or  incorrect.  To  examine  the  difficulty  of  the  items,  we  computed  the  percentage  of 
respondents  who  answered  each  item  correctly;  these  results  are  presented  in  Table  6. 

In  both  years,  the  item  about  flu  shots  appeared  to  be  the  easiest  item  with  80  to  84  percent  of 
respondents  answering  it  correctly.  The  item  concerning  whether  supplemental  insurance  is  the  same  as 
managed  care  was  the  most  difficult  with  the  percentage  of  respondents  answering  correctly  ranging  from 
36  to  43  percent. 

Table  6.  Percentage  of  Correct  Responses  to  the  Three-Item  Quiz  Questions 


Question 

Sample  Member  Interviews 

Proxy  Interviews 

1998 

1999 

1998 

1999 

Medicare  covers  colorectal  cancer  screening 

48% 

50% 

45% 

46% 

Supplemental  insurance  is  the  same  as  managed  care 

43% 

43% 

41% 

36% 

Medicare  covers  an  annual  flu  shot 

84% 

84% 

82% 

80% 

To  help  make  the  difficulty  level  of  the  items  more  meaningful,  we  attempted  to  match  a  grade 
level  to  the  items,  using  information  about  beneficiaries'  educational  achievement.  We  began  by 
classifying  sample  members  in  1999  info  the  following  seven  categories  based  on  the  highest  school 
grade  they  completed:  (1)  no  formal  education,  (2)  8th  grade  or  less,  (3)  9th  to  12th  grade  without  a  high 
school  diploma,  (4)  high  school  graduate,  (5)  some  college,  (6)  bachelor's  degree,  and  (7)  graduate 
degree.  Then  we  calculated  the  percentage  of  sample  members  answering  an  item  correctly,  according  to 
their  educational  achievement.  For  an  item  to  be  matched  to  a  particular  grade  level,  we  expected  at  least 
50%  of  respondents  at  that  grade  level  (as  well  as  at  all  higher  grade  levels)  to  have  answered  the  item 
correctly. 
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Table  7  contains  a  list  of  the  quiz  items,  their  corresponding  grade  level,  and  the  number  of 
respondents  at  that  grade  level  who  answered  the  item  correctly.  As  shown  in  the  table,  the  quiz  question 
about  an  annual  flu  shot  was  very  easy;  71%  of  beneficiaries  with  no  formal  education  answered  this 
question  correctly.  The  question  concerning  supplemental  insurance  was  much  more  difficult;  only  52% 
of  respondents  with  some  college  education  answered  this  question  correctly. 

We  also  computed  a  grade  level  for  the  overall  three-item  quiz.  To  assign  a  grade  level  to  the 
quiz,  we  required  that  at  least  50%  of  respondents  at  that  grade  level  answer  at  least  50%  of  the  quiz 
items  correctly.  In  other  words,  respondents  must  have  received  an  overall  quiz  score  of  2  or  3.  The 
results  indicated  that  the  three-item  quiz  corresponds  to  a  high  school  level.  Fifty-four  percent  of 
beneficiaries  reporting  some  high  school  education  received  a  quiz  score  of  at  least  2. 

Table  7.  Grade  Levels  and  Percentage  of  Correct  Responses  at  Corresponding 
Grade  Level  for  Three-Item  Quiz  Questions 


Question 

Grade  Level 

%  Correct  at 
Grade  Level 

Medicare  covers  colorectal  cancer  screening 

High  school  graduate 

54% 

Supplemental  insurance  is  the  same  as  managed  care 

Some  college 

52% 

Medicare  covers  an  annual  flu  shot 

No  formal  education 

71% 

The  item-total  score  correlations  for  the  questions  in  the  three-item  quiz  are  presented  in 
Table  8.  A  common  criterion  requires  that  correlations  be  greater  than  0.30  for  the  items  to  be 
considered  as  having  contributed  significantly  to  the  scale.  However,  only  one  of  the  items  (colorectal 
cancer  screening)  on  the  three-item  quiz  consistently  met  this  criterion.  In  the  scale  development 
process,  items  with  low  item-total  score  correlations  would  usually  be  removed  from  the  scale;  however, 
given  the  small  number  of  items  on  this  quiz,  it  is  not  feasible  to  remove  any  items. 

Table  8.  Item-Total  Score  Correlations  for  the  Three-Item  Quiz  Questions 


Question 

Sample  Member  Interviews 

Proxy  Interviews 

1998 

1999 

1998 

1999 

Medicare  covers  colorectal  cancer  screening 

0.33 

0.31 

0.37 

0.38 

Supplemental  insurance  is  the  same  as  managed  care 

0.27 

0.26 

0.31 

0.25 

Medicare  covers  an  annual  flu  shot 

0.28 

0.26 

0.28 

0.34 

4.2.2   Scale-Level  Analyses 

Several  scale-level  analyses  were  conducted  on  the  three-item  true/false  quiz.  Specifically,  the 
distribution  of  quiz  scores  was  examined,  and  the  reliability  and  validity  of  the  quiz  were  evaluated. 
Because  the  three-item  quiz  contains  a  small  number  of  possible  scores,  chi-square  analyses  were  used 
for  the  validity  analyses  rather  than  t-tests  and  ANOVAs. 
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Descriptive  Statistics 

Table  9  shows  the  distribution  of  scores  on  the  three-item  quiz.  The  scores  reflect  good 
variability.  In  both  years,  most  respondents  answered  at  least  some  of  the  quiz  items  correctly.  Only 
about  10  percent  of  sample  members  and  13  to  14  percent  of  proxy  respondents  received  the  lowest 
possible  score  (i.e.,  a  score  of  0). 

Table  9.  Frequency  (and  Percentage)  of  Scores  on  the  Three-Item  Quiz 


Sample  Member  Interviews  (Survey  Year) 

Score 

0 

1 

2 

3 

1998  (Round  24) 

1340(11%) 

3642  (29%) 

4387  (35%) 

3225  (26%) 

1999  (Round  27) 

381  (10%) 

1032  (28%) 

1317(36%) 

965  (26%) 

Proxy  Interviews  (Survey  Year) 

0 

1 

2 

3 

1998  (Round  24) 

174(13%) 

435  (32%) 

430(31%) 

331  (24%) 

1999  (Round  27) 

54(14%) 

118(31%) 

125  (33%) 

85  (22%) 

Reliability 

As  a  measure  of  internal  consistency,  Cronbach's  alpha  was  computed  for  the  three  quiz  items. 
The  coefficient  alpha  for  sample  member  respondents  was  0.46  in  1998  and  0.45  in  1999.  Among  proxy 
respondents,  coefficient  alpha  had  a  value  of  0.50  in  both  1998  and  1999.  Values  of  Cronbach's  alpha 
above  0.7  are  usually  considered  acceptable.  Based  on  this  criterion,  these  results  suggest  that  the  three- 
item  quiz  did  not  reach  the  acceptable  level  of  internal  consistency  reliability,  possibly  due  to  the  small 
number  of  items  comprising  the  quiz. 

To  determine  whether  the  internal  consistency  of  the  three-item  quiz  remains  consistent  for 
various  populations,  the  coefficient  alphas  for  the  quiz  were  calculated  separately  for  groups  classified 
according  to  insurance  and  service  utilization  variables.  These  results  are  presented  in  Tables  A-l  1 
through  A-17  of  the  appendix.  The  alphas  were  fairly  similar  for  most  groups.  Among  proxy 
respondents  in  1999,  however,  the  values  for  alphas  varied  widely  across  the  subgroups,  particularly  on 
the  variables  related  to  charges.  For  example,  the  coefficient  alpha  for  those  with  total  reimbursed 
dollars  of  $5,000  or  more  was  0.32,  in  comparison  to  0.60  for  those  with  total  reimbursed  dollars  of  $1  to 
$499.  In  1999,  the  sample  contained  very  few  proxy  respondents  and  among  those,  even  fewer 
individuals  may  have  had  the  highest  level  of  charges,  suggesting  that  perhaps  the  alpha  coefficients  for 
proxy  respondents  in  1999  may  not  provide  good  estimates  of  the  reliability  of  the  quiz  among  those  with 
large  amounts  of  charges. 

Validity 

Chi-square  analyses  revealed  that  sample  members  had  significantly  different  scores  on  the  three- 
item  quiz  than  did  proxy  respondents  in  both  1998  (x2(3)=13.51,  p=.004)  and  1999  (x2(3)=8.67,  p=.  03). 
Therefore,  the  validity  analyses  are  reported  separately  for  these  two  groups  of  respondents. 
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Relationship  with  Global  KANTK  Question 

For  the  first  set  of  validity  analyses,  we  conducted  chi-square  tests  to  examine  the  relationship 
between  the  three-item  quiz  and  the  global  KANTK  question.  The  results  indicated  that  scores  on  the 
three-item  quiz  were  significantly  related  to  responses  to  the  global  KANTK  question  among  sample 
members  during  1998  (x2(12)  =  986.45,  p  <  .0001)  and  1999  (x2(12)  =  302.23,  p  <  .0001).  The  same 
result  was  found  for  proxy  respondents  in  1998  (x2  (12)  =  94.51,  p  <  .0001)  and  1999  (x2  (12)  =  23.28, 
p  =  .025). 

To  help  interpret  the  results  of  these  chi-square  tests,  we  computed  the  frequency  and  percentage 
of  respondents  receiving  each  quiz  score,  according  to  response  to  the  global  KANTK  question  (see 
Tables  10  and  1 1).  For  ease  of  interpretation,  the  percentages  were  calculated  separately  for  each  global 
KANTK  response  category.  For  example,  as  shown  in  Table  10,  38  percent  of  respondents  who 
indicated  that  they  knew  almost  none  of  what  they  needed  to  know  on  the  global  KANTK  question 
received  a  score  of  1  on  the  three-item  quiz  while  only  10  percent  of  those  who  gave  a  response  of 
"almost  none"  received  a  quiz  score  of  3.  The  pattern  of  frequencies  suggests  that  higher  ratings  of 
knowledge  on  the  global  KANTK  question  were  associated  with  higher  scores  on  the  three-item  quiz. 
Generally,  a  higher  percentage  of  beneficiaries  who  responded  "Most"  or  "Just  about  everything" 
received  quiz  scores  of  2  or  3  compared  to  beneficiaries  who  indicated  that  they  knew  "A  little"  or 
"Almost  none"  of  what  they  needed  to  know. 

Table  10.  Frequency  (and  Percentage)  of  Three-Item  Quiz  Scores  by  Response  to 
Global  KANTK  Question  among  Sample  Members 


Sample  Members  - 1998 

3-ltem  Quiz 

Global  KANTK  Question 

Almost 
none 

A  little 

Some 

Most 

Just  about 
everything 

0 

427  (22%) 

351(13%) 

273  (8%) 

164(5%) 

78  (7%) 

1 

747  (38%) 

930  (35%) 

1006(29%) 

660  (22%) 

256  (22%) 

2 

579  (30%) 

912  (34%) 

1294  (37%) 

1106(36%) 

438  (37%) 

3 

204  (10%) 

494(18%) 

948  (27%) 

1123(37%) 

412  (35%) 

Sample  Members  - 1999 

3-ltem  Quiz 

Gl 

obal  KANTK  Q 

uestion 

Almost 
none 

A  little 

Some 

Most 

Just  about 
everything 

0 

140  (23%) 

99(13%) 

71  (7%) 

37  (4%) 

24  (8%) 

1 

214(35%) 

265  (34%) 

299  (28%) 

173  (20%) 

65  (20%) 

2 

184(30%) 

254  (33%) 

410  (39%) 

327  (38%) 

121  (38%) 

3 

82(13%) 

152(20%) 

286  (27%) 

320  (37%) 

110(34%) 
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Table  11.  Frequency  (and  Percentage)  of  Three-Item  Quiz  Scores  by  Response 
to  Global  KANTK  Question  among  Proxy  Respondents 


Proxy  Respondents  - 1998 

3-ltem  Quiz 

Global  KANTK  Question 

Almost 
none 

A  little 

Some 

Most 

Just  about 
everything 

0 

47  (22%) 

45  (15%) 

45  (10%) 

20  (7%) 

8  (7%) 

1 

87  (41%) 

116(39%) 

127  (29%) 

77  (27%) 

18(17%) 

2 

53  (25%) 

83  (28%) 

153(35%) 

89  (32%) 

42  (39%) 

3 

24(11%) 

55(18%) 

110(25%) 

95  (34%) 

41  (38%) 

Proxy  Respor 

idents  - 1999 

3-ltem  Quiz 

Global  KANTK  Question 

Almost 
none 

A  little 

Some 

Most 

Just  about 
everything 

0 

11(17%) 

17  (20%) 

9  (8%) 

10(13%) 

6(16%) 

1 

29  (44%) 

28  (33%) 

34  (30%) 

18(23%) 

6(16%) 

2 

18(27%) 

23  (27%) 

43  (38%) 

25  (33%) 

14  (38%) 

3 

8(12%) 

16(19%) 

26  (23%) 

24(31%) 

1 1  (30%) 

Group  Comparisons 

Next,  to  evaluate  the  validity  of  the  three-item  quiz,  we  conducted  the  known  group  and 
exploratory  comparisons  described  in  Section  3.2.3.  Because  the  three-item  quiz  has  only  four  possible 
scores  (i.e.,  0  to  3),  we  used  chi-square  tests  to  compare  the  quiz  scores  for  the  various  groups;  these 
results  are  presented  separately  by  interview  type  in  Tables  A-18  and  A-19.  Overall,  sample  members 
differed  significantly  on  all  of  the  proposed  variables  during  1998  and  all  of  the  variables  except  for 
managed  care  enrollment  during  1999.  Examining  the  distribution  of  responses  revealed  that  higher  quiz 
scores  were  obtained  by  respondents  between  the  ages  of  65  and  75  years  old,  with  more  education, 
higher  incomes,  private  supplemental  insurance,  any  institutional  or  Part  B  utilization,  some  total  covered 
charges,  total  reimbursed  dollars,  Part  B  charges,  or  covered  institutional  charges.  In  addition,  sample 
members  in  1998  who  had  been  enrolled  in  managed  care  during  the  past  year  had  higher  three-item  quiz 
scores. 

As  shown  in  Table  A-19,  proxy  respondents  differed  on  most  of  the  proposed  variables.  The 
response  distributions  suggest  that  higher  three-item  quiz  scores  were  associated  with  respondents  who 
had  more  education,  higher  incomes,  private  supplemental  insurance,  any  institutional  or  Part  B 
utilization,  some  total  covered  charges,  total  reimbursed  dollars,  Part  B  charges,  or  covered  institutional 
charges.    In  addition,  among  proxies  in  1999,  those  who  had  been  enrolled  in  managed  care  during  the 
past  year  had  higher  three-item  quiz  scores. 
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4.3     Seven-Item  Quiz 

4.3.1    Item-Level  Analyses 

Table  12  displays  the  percentages  of  correct  responses  to  the  questions  that  comprise  the  seven- 
item  quiz.  Across  both  years,  the  question  that  asks  whether  HMOs  cover  more  health  services  was  the 
most  difficult  item,  with  a  percentage  correct  ranging  from  35  percent  to  40  percent.  The  question  about 
whether  Medicare  alone  pays  for  all  health  care  expenses  received  the  highest  percentage  of  correct 
responses,  indicating  that  it  was  the  easiest  item  for  respondents,  possibly  because  it  addresses  a 
particularly  relevant  topic  for  beneficiaries. 


Table  12.  Percentage  of  Correct  Responses  to  the  Seven-Item  Quiz  Questions 


Question 

Sample  Member  Interviews 

Proxy  Interviews 

1998 

1999 

1998 

1999 

Can  select  different  health  plan  options 

47% 

44% 

45% 

43% 

Medicare  alone  pays  for  all  health  care  expenses 

76% 

78% 

69% 

76% 

Medicare  offers  more  information 

45% 

52% 

44% 

50% 

Can  report  complaints  to  Medicare  about  HMOs 
and  supplemental  insurance 

58% 

56% 

56% 

56% 

Limited  choices  doctors  if  on  HMOs 

63% 

61% 

59% 

54%  * 

Can  drop  HMO  and  still  be  covered  by  Medicare 

47% 

46% 

43% 

39% 

HMOs  cover  more  health  services 

40% 

37% 

39% 

35% 

We  also  conducted  grade-level  analyses  for  both  the  individual  quiz  items  and  the  overall  seven- 
item  quiz.  The  grade  level  and  corresponding  percentage  correct  for  the  seven-item  quiz  questions  are 
presented  in  Table  13.  As  shown  in  the  table,  the  seven-item  quiz  contains  items  covering  all  education 
levels  except  for  no  formal  education.  The  item  concerning  whether  Medicare  alone  pays  for  all 
expenses  was  easiest  with  65%  of  those  having  an  8th  grade  education  or  less  answering  it  correctly.  In 
contrast,  the  item  about  whether  HMOs  cover  more  health  services  was  very  difficult;  only  50%  of 
respondents  with  a  graduate  degree  answered  it  correctly. 

To  assign  a  grade  level  to  the  seven-item  quiz,  at  least  50%  of  beneficiaries  at  the  grade  level 
must  have  answered  at  least  50%  of  the  questions  correctly  (i.e.,  received  a  quiz  score  of  4  or  higher). 
Similar  to  the  three-item  quiz,  the  seven-item  quiz  corresponds  to  a  high  school  level  of  difficulty.  Fifty- 
one  percent  of  sample  members  with  some  high  school  education  received  an  overall  quiz  score  of  4  or 
higher. 
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Table  13.  Grade  Levels  and  Percentage  of  Correct  Responses  at  Corresponding 
Grade  Level  for  Seven-Item  Quiz  Questions 


Question 

Grade  Level 

%  Correct  at 
Grade  Level 

Can  select  different  health  plan  options 

Bachelor's  degree 

53% 

Medicare  alone  pays  for  all  health  care  expenses 

8    grade  or  less 

65% 

Medicare  offers  more  information 

High  school  graduate 

55% 

Can  report  complaints  to  Medicare  about  HMOs  and 
supplemental  insurance 

Some  high  school 

52% 

Limited  choices  doctors  if  on  HMOs 

Some  high  school 

59% 

Can  drop  HMO  and  still  be  covered  by  Medicare 

Some  college 

48% 

HMOs  cover  more  health  services 

Graduate  degree 

50% 

Table  14  shows  the  item-total  score  correlations  for  the  questions  on  the  seven-item  quiz.  All  of 
the  items  had  item-total  score  correlations  over  0.30,  suggesting  that  they  are  all  contributing  to  the 
internal  consistency  reliability  of  the  scale.  The  four  items  concerning  HMOs  had  the  highest 
correlations,  indicating  that  they  are  most  related  to  the  underlying  construct  measured  by  the  scale.  Of 
these  four  items,  the  item  that  asks  whether  beneficiaries  can  drop  an  HMO  and  still  be  covered  by 
Medicare  had  the  highest  item-total  score  correlations.  Among  all  seven  items  on  the  quiz,  the  two  items 
about  whether  Medicare  alone  pays  for  all  health  care  expenses  and  whether  Medicare  offers  more 
information  appeared  to  be  the  least  related  to  the  underlying  construct. 

Table  14.  Item-Total  Score  Correlations  for  the  Seven-Item  Quiz  Questions 


Question 

Sample  Member  Interviews 

Proxy  Interviews 

1998 

1999 

1998 

1999 

Can  select  different  health  plan  options 

0.44 

0.40 

0.44 

0.44 

Medicare  alone  pays  for  all  health  care  expenses 

0.35 

0.38 

0.38 

0.40 

Medicare  offers  more  information 

0.37 

0.37 

0.38 

0.38 

Can  report  complaints  to  Medicare  about  HMOs  and 
supplemental  insurance 

0.46 

0.43 

0.52 

0.50 

Limited  choices  of  doctors  if  on  HMOs 

0.52 

0.49 

0.58 

0.52 

Can  drop  HMO  and  still  be  covered  by  Medicare 

0.55 

0.53 

0.60 

0.55 

HMOs  cover  more  health  services 

0.49 

0.46 

0.51 

0.49 

4.3.2    Scale-Level  Analyses 

Following  the  item-level  analyses,  additional  analyses  were  conducted  to  explore  the 
psychometric  properties  of  the  seven-item  quiz  as  a  whole.  This  section  describes  the  results  of  these 
scale-level  analyses. 
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Descriptive  Statistics 

Table  15  shows  the  descriptive  statistics  for  the  seven-item  quiz  scores,  displayed  separately  for 
sample  member  and  proxy  respondents.  As  shown  in  the  table,  the  distribution  of  scores  was  slightly 
skewed  to  the  right  with  the  mode  generally  having  a  higher  value  than  the  mean  and  median.  Most 
respondents  received  scores  of  "4"  or  "5,"  indicating  that  they  correctly  answered  more  than  half  of  the 
questions. 

Table  15.  Descriptive  Statistics  for  the  Seven-Item  Quiz 


Sample  Member  Interviews  (Survey  Year) 

N 

Mean 

S.D. 

Median 

Mode 

1998  (Round  24) 

13,062 

3.76 

2.12 

4 

5 

1999  (Round  27) 

3,920 

3.74 

2.08 

4 

4 

Proxy  Interviews  (Survey  Year) 

N 

Mean 

S.D. 

Median 

Mode 

1998  (Round  24) 

1,470 

3.56 

2.22 

4 

5 

1999  (Round  27) 

418 

3.54 

2.16 

4 

5 

Reliability 

The  internal  consistency  reliability  of  the  seven-item  quiz  was  measured  using  Cronbach's  alpha. 
The  alpha  coefficient  for  the  seven-item  quiz  was  0.74  for  sample  members  in  1998  and  0.73  for  sample 
members  in  1999.  Among  proxy  respondents,  the  alpha  coefficient  for  the  quiz  was  0.77  in  1998  and 
0.75  in  1999.  These  values  indicate  that  the  seven-item  quiz  reached  an  acceptable  level  of  reliability. 

As  mentioned  in  section  2.0,  there  was  an  additional  quiz  item  available  in  1998.  When  this  item 
was  included  in  the  calculation  of  the  quiz  scores  during  1998,  the  quiz  had  an  alpha  of  0.76  for  sample 
members  and  0.79  for  proxy  respondents  (Bann  et  al.,  2000).  These  alpha  coefficients  are  very  similar  to 
those  found  here  for  the  seven-item  quiz,  suggesting  that  the  removal  of  this  additional  item  did  not  have 
a  significant  impact  on  the  internal  consistency  of  the  quiz. 

The  alpha  coefficients  for  the  seven-item  quiz  were  also  calculated  separately  for  various 
subgroups  to  assess  whether  the  reliability  of  the  quiz  remained  consistent.  The  results  are  presented  in 
Tables  A-20  through  A-26  of  the  appendix.  Generally,  the  values  of  alpha  were  very  consistent  across  all 
of  the  groups,  with  alphas  ranging  from  0.69  to  0.78  for  sample  members  and  from  0.71  to  0.79  for  proxy 
respondents. 

Validity 

To  assess  construct  validity,  particular  groups  who  were  expected  to  differ  in  knowledge  were 
compared  to  determine  if  they  did  in  fact  receive  different  scores  on  the  seven-item  quiz.  (See 
Tables  A-27  and  A-28  in  the  appendix  for  details  of  the  statistical  results.)  These  analyses  were 
computed  separately  for  sample  member  and  proxy  respondents.  As  hypothesized,  sample  members 
received  significantly  higher  scores  on  the  seven-item  quiz  in  1998  than  did  proxy  respondents 
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(t(1786)  =  3.30,  p  =  .001).  However,  the  1999  quiz  scores  of  sample  members  did  not  differ  significantly 
from  those  of  proxy  respondents  (t(4336)  =  1.81,  n.s.). 

Relationship  with  Global  KANTK  Question 

ANOVAs  revealed  that  the  seven-item  quiz  scores  of  sample  members  varied  significantly 
according  to  responses  to  the  global  KANTK  question  in  both  1998  (F(4,  12972)  =  348.68,  p  <  .0001) 
and  1999  (F(4,  3909)  =  1 19.86,  p  <  .0001).  Similar  results  were  found  for  proxy  respondents  in  1998 
(F(4,  1455)  =  39.70,  p  <  .0001)  and  1999  (F(4, 411)  =  9.37,  p  <  .0001). 

Table  16  displays  the  means  and  standard  deviations  of  the  seven-item  quiz  scores  by  responses 
to  the  global  KANTK  question.  For  both  years  and  both  types  of  respondents,  there  is  a  monotonic 
relationship  between  scores  on  the  seven-item  quiz  and  the  first  four  response  categories  ("Almost  none," 
"A  little,"  "Some,"  and  "Most")  for  the  global  KANTK  question.  Respondents  who  rated  their 
knowledge  higher  on  the  global  KANTK  question  received  higher  scores  on  the  seven-item  quiz.  This 
pattern  seems  to  taper  off  for  the  last  response  category  ("Just  about  everything").  Generally, 
respondents  with  ratings  of  "Most"  or  "Just  about  everything"  have  similar  quiz  scores.  It  is  possible  that 
respondents  do  not  clearly  distinguish  between  the  "Most"  and  "Just  about  everything"  response 
categories. 

Table  16.    Means  (and  Standard  Deviations)  of  Seven-Item  Quiz  Scores  by 
Response  to  Global  KANTK  Question 


Sample  Member  Interviews 
(Survey  Year) 

Almost  none 

A  little 

Some 

Most 

Just  about 
everything 

1998  (Round  24) 

2.5(2.1) 

3.4(2.1) 

3.9(2.0) 

4.5(1.9) 

4.4  (2.0) 

1999  (Round  27) 

2.5  (2.0) 

3.4  (2.0) 

3.9(1.9) 

4.4(1.8) 

4.4(2.1) 

Proxy  Interviews  (Survey  Year) 

Almost  none 

A  little 

Some 

Most 

Just  about 
everything 

1998  (Round  24) 

2.3  (2.2) 

3.2(2.2) 

3.8  (2.0) 

4.3  (2.0) 

4.3(2.1) 

1999  (Round  27) 

2.4(1.8) 

3.2  (2.2) 

3.9(2.1) 

4.3(1.9) 

3.8  (2.4) 

Group  Comparisons 

ANOVAs  and  t-tests  were  conducted  to  determine  whether  various  subgroups  received  different 
scores  on  the  seven-item  quiz.  The  mean  seven-item  quiz  scores  for  each  of  the  groups  are  presented 
separately  according  to  interview  type  in  Tables  A-27  and  A-28.  Asterisks  indicate  the  significance  level 
of  the  corresponding  ANOVA  or  t-test. 


2  Please  note  that  for  these  t-tests  the  degrees  of  freedom  are  smaller  for  the  1998  sample  then  the  1999  sample  even 
though  it  has  a  larger  sample  size.  The  Folded  F  test  for  equality  of  variances  indicated  that  the  variances  for  the 
proxy  and  sample  members  in  1998  could  not  be  assumed  to  be  equal.  Therefore,  Satterthwaite's  approximation 
was  used  which  usually  results  in  a  decrease  in  the  degrees  of  freedom. 
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As  shown  in  Table  A-27,  seven  item  quiz  scores  for  sample  members  in  1998  differed 
significantly  on  all  of  the  variables  except  for  institutional  utilization.  The  results  for  sample  members  in 
1999  were  significant  for  all  variables  except  institutional  utilization,  Part  B  utilization,  and  covered 
institutional  charges.  The  patterns  of  mean  scores  for  both  1998  and  1999  indicate  that  the  results  are 
generally  in  the  expected  direction.  Higher  quiz  scores  were  associated  with  sample  members  who  had 
higher  education,  higher  incomes,  some  charges,  private  supplemental  insurance,  or  were  between  the 
ages  of  65  and  75.  In  addition,  as  hypothesized,  respondents  who  had  been  enrolled  in  managed  care 
during  the  past  year  had  higher  seven-item  quiz  scores  than  those  who  had  not  been  enrolled  during  the 
past  year. 

Table  A-28  displays  the  results  for  proxy  respondents  in  1998  and  1999.  There  were  fewer 
significant  results  for  proxy  respondents  than  sample  members,  particularly  during  1999.  Across  both 
years,  higher  quiz  scores  were  received  by  respondents  with  more  education,  higher  incomes,  private 
supplemental  insurance,  or  who  had  been  enrolled  in  managed  care  during  the  past  year.  During  1 998, 
higher  scores  were  also  found  for  respondents  with  some  Part  B  utilization  and  any  allowed  Part  B 
charges. 
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We  developed  three  potential  knowledge  measures  using  data  from  the  Beneficiary  Knowledge 
(BK)  and  Beneficiary  Needs  (BN)  supplemental  rounds  of  the  CY  1998  and  1999  Medicare  Current 
Beneficiary  Survey  (MCBS).  The  first  measure,  the  perceived  knowledge  index,  includes  five  questions 
that  ask  beneficiaries  to  rate  how  much  they  know  about  a  particular  topic  related  to  Medicare.  The  other 
two  measures  are  a  three-item  and  a  seven-item  quiz  that  require  participants  to  respond  to  a  series  of 
true/false  questions. 

We  evaluated  the  psychometric  properties  of  the  knowledge  measures  by  calculating  item-level 
and  scale-level  descriptive  statistics,  assessing  internal  consistency  reliability,  and  conducting  construct 
validity  analyses.  The  item-level  analyses  of  the  perceived  knowledge  items  suggest  that  these  items 
appropriately  do  not  exhibit  floor  or  ceiling  effects.  Comparisons  of  the  3-item  and  7-item  quiz  scores 
according  to  beneficiaries'  educational  achievement  suggest  that  the  difficulty  levels  of  both  of  these 
quizzes  are  equivalent  to  a  high  school  level. 

To  assess  internal  consistency  reliability,  we  computed  Cronbach's  alphas  for  each  of  the 
measures.  A  level  of  0.70  is  commonly  required  for  an  alpha  to  be  considered  acceptable  for  use  in 
group-level  analyses.  As  shown  in  Table  17,  both  the  seven-item  quiz  and  the  perceived  knowledge 
index  met  this  criterion.  The  seven-item  quiz  had  coefficient  alphas  of  0.73  to  0.77,  while  the  perceived 
knowledge  index  demonstrated  strong  internal  consistency  reliability  with  alphas  ranging  from  0.82  to 
0.85.  With  Cronbach's  alphas  ranging  from  0.45  to  0.50,  the  three-item  quiz  did  not  meet  the 
acceptability  criterion.  These  low  coefficient  alphas  may  be  due  to  the  small  number  of  items  on  the 
quiz,  suggesting  that  the  internal  consistency  reliability  of  the  three-item  quiz  could  be  increased  by 
adding  items. 

Table  17.    Coefficient  Alphas  of  the  Knowledge  Indices 


Sample  Member  Interviews 
(Survey  Year) 

Perceived 

Knowledge 

Index 

3-ltem 
Quiz 

7-ltem 
Quiz 

1998  (Round  24) 

.82 

.46 

.74 

1999  (Round  27) 

.82 

.45 

.73 

Proxy  Interviews  (Survey. Year) 

1998  (Round  24) 

.84 

.50 

.77 

1999  (Round  27) 

.85 

.50 

.75 
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To  evaluate  the  construct  validity  of  the  knowledge  indices,  we  examined  the  relationship 
between  each  index  and  a  global  knowledge  item  that  asks  respondents  how  much  they  feel  they  know 
about  the  Medicare  program,  referred  to  as  the  global  know-all-need-to-know  (KANTK)  question.  A 
strong  relationship  between  a  knowledge  index  and  another  measure  of  the  same  construct  (e.g.,  the 
global  KANTK  question)  would  provide  support  for  the  construct  validity  of  the  knowledge  index.  The 
results  indicated  that  all  three  knowledge  indices  are  significantly  related  to  the  global  KANTK  question. 
Respondents  who  rated  their  knowledge  higher  on  the  global  KANTK  question  received  higher  scores  on 
each  knowledge  index.  In  particular,  there  appeared  to  be  a  strong,  monotonic  relationship  between  the 
perceived  knowledge  index  and  the  global  KANTK  question,  possibly  because  of  the  similarity  in 
wording  between  the  two.  Nonetheless  we  can  infer  that  they  closely  measure  the  same  construct. 

As  a  further  evaluation  of  the  validity  of  the  knowledge  indices,  we  analyzed  comparisons  of  the 
mean  knowledge  index  scores  for  various  groups.  The  first  set  of  analyses,  sometimes  referred  to  as 
known-groups  comparisons,  involve  comparisons  between  groups  previously  shown  to  have  different 
knowledge  levels.  We  also  compared  groups  we  hypothesized  would  differ  in  knowledge  which  we 
referred  to  as  exploratory  comparisons. 

All  three  knowledge  indices  performed  well  on  the  known  group  comparisons.  In  general, 
groups  that  were  expected  to  differ  in  knowledge  had  significantly  different  knowledge  index  scores. 
Because  of  the  small  number  of  proxy  respondents  in  1999,  however,  the  knowledge  index  scores  for  this 
group  had  much  lower  statistical  power  and  the  differences  were  therefore  less  likely  to  be  significant. 

One  of  the  most  consistent  findings  across  all  years  and  interview  types  was  a  strong  relationship 
between  higher  educational  achievement  and  higher  knowledge  scores.  As  mentioned  in  Chapter  3,  this 
relationship  has  been  found  repeatedly  in  prior  research.  Possible  explanations  include  those  with  more 
education  having  better  comprehension  of  the  program  materials  or  a  greater  motivation  to  learn  about 
the  Medicare  program.  Another  finding  was  that,  as  hypothesized,  the  relationship  between  knowledge 
scores  and  enrollment  in  managed  care  was  strongest  for  the  seven-item  quiz.  This  quiz  includes  four 
questions  concerning  managed  care  plans,  while  the  three-item  quiz  and  the  perceived  knowledge  index 
each  contain  only  one  question  on  this  topic. 

Based  solely  on  the  quantitative  results,  the  perceived  knowledge  index  seems  to  have  the  best 
psychometric  properties.  This  index  performed  well  in  both  the  reliability  and  validity  analyses. 
However,  other  criteria,  such  as  the  content  of  the  items,  must  be  considered  when  selecting  the  most 
appropriate  measure  of  knowledge.  The  perceived  knowledge  index  relies  on  beneficiaries  to  be  the  sole 
judge  of  their  knowledge.  Individuals'  subjective  ratings  of  their  own  knowledge  may  be  influenced  by 
factors  other  than  knowledge,  such  as  confidence  in  decision-making  or  satisfaction  with  information 
received.  Therefore,  the  perceived  knowledge  index  may  not  provide  the  most  accurate  assessment  of 
actual  knowledge  level.  A  more  precise  measure  of  knowledge  would  require  respondents  to 
demonstrate  their  knowledge,  such  as  by  correctly  answering  the  true/false  questions  included  on  the 
three-item  and  seven-item  quizzes. 
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On  the  basis  of  both  content  considerations  and  the  psychometric  analysis  results,  the  seven-item 
quiz  appears  to  be  the  most  useful  measure  of  beneficiary  knowledge  overall.  As  mentioned  above,  the 
quiz  requires  beneficiaries  to  demonstrate  their  knowledge  rather  than  merely  stating  that  they  know 
everything  they  need  to  know.  In  addition,  the  quiz  had  good  variability  in  scores,  reached  an  acceptable 
level  of  internal  consistency  reliability,  and  performed  well  on  the  validity  analyses. 

Possible  modifications  to  the  seven-item  quiz  depend  on  the  intended  use  of  the  measure.  If  the 
goal  of  the  quiz  is  to  obtain  the  most  precise  estimate  of  beneficiaries'  knowledge  possible,  then  the  quiz 
should  contain  questions  that  cover  all  difficulty  levels.  The  grade-level  analyses  suggest  that  the  items 
on  the  seven-item  quiz  cover  all  education  levels  except  for  no  formal  education.  Very  few  respondents 
report  having  no  formal  education  and  therefore  it  may  not  be  necessary  to  add  questions  specifically 
targeted  at  this  group.  However,  the  quiz  contains  only  one  item  at  the  high  school  graduate  level. 
Because  more  respondents  report  being  a  high  school  graduate  than  any  other  educational  level,  it  may  be 
helpful  to  add  more  items  targeted  at  this  group. 

In  contrast,  if  the  purpose  of  the  quiz  is  to  determine  whether  Medicare  beneficiaries  reach  a 
certain  proficiency  level,  then  the  primary  emphasis  should  be  on  the  content  of  the  quiz  items.  The 
items  should  cover  the  entire  range  of  information  that  beneficiaries  need  to  know  in  order  to  make 
informed  decisions.  Less  emphasis  should  be  placed  on  the  difficulty  of  the  items.  In  this  case,  a 
limitation  of  the  seven-item  quiz  is  that  four  of  its  seven  questions  deal  with  managed  care  plans  which 
restricts  the  range  of  knowledge  that  the  quiz  can  measure.  Other  questions  could  be  added  to  improve 
the  comprehensiveness  of  the  quiz.  In  fact,  as  part  of  RTFs  Questionnaire  Development  and  Cognitive 
Testing  Using  Item  Response  Theory  project,  several  new  knowledge  questions  were  developed  that 
address  a  variety  of  issues,  including  beneficiary  rights  and  health  plan  decision-making  (Unrig  et  al, 
2001),  and  could  be  used  to  expand  the  seven-item  quiz. 
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Table  A-1.  MCBS  Knowledge  Questions 


Question  Wording 


Perceived  Knowledge  Index 


Round 


24,27 


Year 


1998, 1999 


BN  1 :  How  much  do  you  feel  you  know  about  what  medical  services  Medicare  covers  or  does  not  cover?  Do  you  know  just  about 
everything  you  need  to  know,  most  of  what  you  need  to  know,  some  of  what  you  need  to  know,  a  little  of  what  you  need  to  know,  or 
almost  none  of  what  you  need  to  know  about  what  Medicare  covers  or  doesn't  cover? 


BN  2:  How  much  do  you  feel  you  know  about  how  much  you  have  to  pay  for  medical  services?  Do  you  know  just  about  everything 
you  need  to  know,  most  of  what  you  need  to  know,  some  of  what  you  need  to  know,  a  little  of  what  you  need  to  know,  or  almost  none 
of  what  you  need  to  know  about  how  much  you  have  to  pay  for  medical  services? 


BN  3:  How  much  do  you  feel  you  know  about  supplemental  or  Medigap  insurance,  such  as  what  it  covers  or  how  it  works  with 
Medicare  to  pay  medical  claims?  Do  you  know  just  about  everything  you  need  to  know,  most  of  what  you  need  to  know,  some  of  what 
you  need  to  know,  a  little  of  what  you  need  to  know,  or  almost  none  of  what  you  need  to  know  about  supplemental  insurance? 

BN  4:  How  much  do  you  feel  you  know  about  the  availability  and  benefits  of  Medicare  managed  care  plans?  Do  you  know  just  about 
everything  you  need  to  know,  most  of  what  you  need  to  know,  some  of  what  you  need  to  know,  a  little  of  what  you  need  to  know,  or 
almost  none  of  what  you  need  to  know  about  the  availability  and  benefits  of  managed  care  plans? 

BN  5:  How  much  do  you  feel  you  know  about  choosing  or  finding  a  doctor  or  other  health  care  provider?  Do  you  know  just  about 
everything  you  need  to  know,  most  of  what  you  need  to  know,  some  of  what  you  need  to  know,  a  little  of  what  you  need  to  know,  or 
almost  none  of  what  you  need  to  know  about  finding  a  doctor  or  other  health  care  provider? 


Three-Item  Quiz1,2 


24,27        1998, 1999 


BN  16:  Medicare  covers  colorectal  cancer  screening.  (True) 


BN  18:  Medigap  or  supplemental  insurance  is  the  same  as  a  Medicare  managed  care  plan.  (False) 
BN  19:  Medicare  covers  an  annual  flu  shot.  (True) 


Source:  Medicare  Current  Beneficiary  Survey  Supplemental  Rounds  BN  24  and  27. 


Table  A-1  (continued) 


> 

i 


Question  Wording 


Seven-Item  Quiz1,2 


Round 


23,26 


Year 


1998, 1999 


BK  43:  Most  people  covered  by  Medicare  can  select  among  different  kinds  of  health  plan  options  within  Medicare.  (True) 

BK  44:  Medicare  without  a  supplemental  insurance  policy  pays  for  all  of  your  health  care  expenses.  (False) ; 

BK  46:  The  Medicare  program  has  begun  to  offer  more  information  and  help  in  order  to  answer  your  Medicare  questions.  (True) 

BK  47:  People  can  report  complaints  to  Medicare  about  their  Medicare  managed  care  plans  (HMOs)  or  supplemental  plans  if  they  are 
not  satisfied  with  them.  (True) . . 


BK  48:  If  someone  joins  a  Medicare  managed  care  plan  (HMO)  that  covers  people  on  Medicare,  they  have  limited  choices  about  which 
doctors  they  can  see.  (True) . 


BK  49:  If  someone  joins  a  Medicare  managed  care  plan  (HMO)  that  covers  people  on  Medicare,  they  can  change  or  drop  the  plan  and 
still  be  covered  by  Medicare.  (True) 


BK  50:  Medicare  managed  care  plans  (HMOs)  that  cover  people  on  Medicare  often  cover  more  health  services,  like  prescribed 
medicines,  than  Medicare  without  a  supplemental  policy.  (True) 


Source:  Medicare  Current  Beneficiary  Survey  Supplemental  Rounds  BK  23  and  26 
'  The  correct  answers  to  the  quiz  questions  are  presented  in  italics  following  each  question. 

2  For  each  of  the  quizzes,  only  one  of  the  questions  has  a  correct  answer  of  false.  Because  there  is  a  possibility  of  a  response  set  bias,  it  is  often  preferable  to 
vary  the  correct  responses  to  the  questions  on  a  quiz  rather  than  having  the  same  correct  response  for  almost  all  of  the  questions. 


Table  A-2.  Coefficient  Alphas  of  the  Perceived  Knowledge  Index,  by  Enrollment 
in  Managed  Care  During  the  Past  Year 


Interview  Type/Survey  Year 

No  Enrollment 

Some  Enrollment 

Sample  Member  Interviews 

1998  (Round  24) 

0.82 

0.83 

1999  (Round  27) 

0.83 

0.84 

Proxy  Interviews 

1998  (Round  24) 

0.84 

0.85 

1999  (Round  27) 

0.85 

0.87 

SOURCE:  Centers  for  Medicare  &  Medicaid  Services,  Medicare  Current  Beneficiary  Survey  1998  and 
1999  Access  to  Care  and  Supplemental  Files. 


Table  A-3.  Coefficient  Alphas  of  the  Perceived  Knowledge  Index,  by  Institutional 
Utilization1 


Interview  Type/Survey  Year 

No  Utilization 

Some  Utilization 

Sample  Member  Interviews 

1998  (Round  24) 

0.83 

0.82 

1999  (Round  27) 

0.83 

0.82 

Proxy  Interviews 

1998  (Round  24) 

0.86 

0.83 

1999  (Round  27) 

0.87 

0.84 

Only  respondents  who  were  not  enrolls 

;d  in  managed  care  during  the  past  y 

ear  were  included  in  these  analyses. 

Table  A-4.  Coefficient  Alphas  of  the  Perceived  Knowledge  Index,  by  Part  B  Utilization1 


Interview  Type/Survey  Year 

No  Utilization 

Some  Utilization 

Sample  Member  Interviews 

1998  (Round  24) 

0.83 

0.82 

1999  (Round  27) 

0.83 

0.82 

Proxy  Interviews 

1998  (Round  24) 

0.87 

0.84 

1999  (Round  27) 

0.87 

0.85 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses 
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Table  A-5.  Coefficient  Alphas  of  the  Perceived  Knowledge  Index,  by  Total 
Covered  Charges1 


Interview  Type/Survey  Year 

$0 

$1  -  $499 

$500  -  $4,999 

$5,000  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.83 

0.82 

0.82 

0.83 

1999  (Round  27) 

0.83 

0.82 

0.82 

0.83 

Proxy  Interviews 

1998  (Round  24) 

0.87 

0.84 

0.84 

0.82 

1999  (Round  27) 

0.86 

0.86 

0.85 

0.83 

Table  A-6.  Coefficient  Alphas  of  the  Perceived  Knowledge  Index,  by  Total 
Reimbursed  Dollars1 


Interview  Type/Survey  Year 

$0 

$1  -  $499 

$500  -  $4,999 

$5,000  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.83 

0.82 

0.82 

0.83 

1999  (Round  27) 

0.83 

0.82 

0.82 

0.83 

Proxy  Interviews 

1998  (Round  24) 

0.86 

0.85 

0.83 

0.82 

1999  (Round  27) 

0.87 

0.86 

0.84 

0.83 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses 


Table  A-7.  Coefficient  Alphas  of  the  Perceived  Knowledge  Index,  by  Covered 
Institutional  Charges1 


Interview  Type/Survey  Year 

$0 

$1  -  $499 

$500  -  $4,999 

$5,000  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.83 

0.82 

0.82 

0.83 

1999  (Round  27) 

0.83 

0.81 

0.83 

0.83 

Proxy  Interviews 

1998  (Round  24) 

0.86 

0.83 

0.84 

0.81 

1999  (Round  27) 

0.87 

0.85 

0.84 

0.82 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
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Table  A-8.  Coefficient  Alphas  of  the  Perceived  Knowledge  Index,  by  Allowed 
Part  B  Charges1 


Interview  Type/Survey  Year           $0 

$1  -  $499 

$500 -$1,499 

$1,500  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.83 

0.83 

0.82 

0.82 

1999  (Round  27) 

0.83 

0.83 

0.82 

0.82 

Proxy  Interviews 

1998  (Round  24) 

0.87 

0.83 

0.84 

0.83 

1999  (Round  27) 

0.87 

0.86 

0.84 

0.83 

1 0nly  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  includec 

in  these  analyses. 
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Table  A-9.  Mean  Perceived  Knowledge  Index  Scores  and  Statistical 

Significance  of  ANOVAs  and  t-Tests  among  Sample  Members 


Variable 

Sample  Members 
1998 

Sample  Members 
1999 

Education 

8th  grade  or  less 

12.92*** 

13.06*** 

More  than  8th  grade,  but  no  college 

15.28 

15.54 

College 

16.59 

17.19 

Age  category 

65-75 

15.84*** 

16.31*** 

Over  75 

15.07 

15.54 

Income  category 

Under  $25,000 

14.50*** 

14.84*** 

$25,000  or  more 

16.86 

17.24 

Managed  care 

Some  enrollment 

16.37*** 

16.71*** 

No  enrollment 

14.95 

15.38 

Private  Supplemental  Insurance 

Supplemental  insurance 

16.03*** 

16.25*** 

No  supplemental  insurance 

14.62 

14.75 

Any  institutional  utilization  ' 

Some  utilization 

15.07*** 

15.61*** 

No  utilization 

14.72 

14.90 

Any  Part  B  utilization  ' 

Some  utilization 

15.17*** 

15.58*** 

No  utilization 

13.28 

13.63 

Total  covered  charges  * 

$0 

13.89*** 

13.69*** 

$l-$499 

14.70 

15.06 

$500-$4,999 

15.38 

15.76 

$5,000  or  more 

15.12 

15.62 

Total  reimbursed  dollars ' 

■     ■                             ■■■..-■■" 

$0 

11.95*** 

13.84*** 

$l-$499 

13.52 

15.34 

$500-$4,999 

14.97 

15.86 

$5,000  or  more 

15.43 

15.38 

Covered  institutional  charges  ! 

';                                                ^     ;  ; 

$0 

14.72*** 

14.90*** 

$l-$499 

15.05 

15.72 

$500-$4,999 

15.11 

15.61 

$5,000  or  more 

15.04 

15.50 

Allowed  Part  B  charges  ' 

$0 

13.34*** 

13.77*** 

$l-$499 

14.75 

15.06 

$500-$  1,499 

15.37 

15.84 

$1,500  or  more 

15.44 

15.81 

1 0nly  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
*  p  <  .05,  **  p  <  .01,  ***  p  <  .001 


A-6 


Table  A-10.  Mean  Perceived  Knowledge  Index  Scores  and  Statistical 

Significance  of  ANOVAs  and  t-Tests  among  Proxy  Respondents 


Variable 

Proxy  Respondents 
1998 

Proxy  Respondents 
1999 

Education 

8th  grade  or  less 

13.88*** 

14.36*** 

More  than  8th  grade,  but  no  college 

15.22 

15.36 

College 

16.17 

16.37 

Age  category 

65-75 

15.38 

15.90 

Over  75 

15.09 

15.48 

Income  category 

Under  $25,000 

14.28*** 

14.63*** 

$25,000  or  more 

16.85 

17.13 

Managed  care 

'      *      '                              '    .           ■        ■         ^         '      ■                                    ■■;■'■''                 \ 

Some  enrollment 

15.42 

15.52 

No  enrollment 

14.70 

15.09 

Private  Supplemental  Insurance 

Supplemental  insurance 

15  99*** 

15.64 

No  supplemental  insurance 

14.27 

14.23 

Any  institutional  utilization 

Some  utilization 

14.98* 

15.30 

No  utilization 

14.24 

14.70 

Any  Part  B  utilization  ! 

Some  utilization 

14  97*** 

15.32*** 

No  utilization 

13.25 

13.72 

Total  covered  charges ' 

$0 

13.60*** 

13.83** 

$l-$499 

14.31 

14.86 

$500-$4,999 

15.10 

15.19 

$5,000  or  more 

15.02 

15.68 

Total  reimbursed  dollars ' 

$0 

13.55*** 

13.93** 

$l-$499 

14.55 

14.88 

$500-$4,999 

15.24 

15.35 

$5,000  or  more 

14.91 

15.82 

Covered  institutional  charges ' 

$0 

14.23* 

14.70* 

$l-$499 

14.90 

14.90 

$500-$4,999 

15.36 

14.96 

$5,000  or  more 

14.67 

15.91 

Allowed  Part  B  charges  ' 

$0 

13.45*** 

13.62*** 

$l-$499 

14.74 

14.85 

$500-$  1,499 

14.83 

15.29 

$1,500  or  more 

15.22 

15.88 

1 0nly  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
p<.05,  **p<.01,***p<.001 
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Table  A-1 1 .   Coefficient  Alphas  of  the  Three-Item  Quiz,  by  Enrollment  in 
Managed  Care  During  the  Past  Year 


Interview  Type/Survey  Year 

No  Enrollment 

Some  Enrollment 

Sample  Member  Interviews 

1998  (Round  24) 

0.47 

0.41 

1999  (Round  27) 

0.46 

0.40 

Proxy  Interviews 

1998  (Round  24) 

0.51 

0.43 

1999  (Round  27) 

0.52 

0.38 

Table  A-1 2.  Coefficient  Alphas  of  the  Three-Item  Quiz,  by  Institutional 
Utilization1 


Interview  Type/Survey  Year 

No  Utilization 

Some  Utilization 

Sample  Member  Interviews 

1998  (Round  24) 

0.50 

0.45 

1999  (Round  27) 

0.48 

0.44 

Proxy  Interviews 

1998  (Round  24) 

0.53 

0.48 

1999  (Round  27) 

0.54 

0.49 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 


Table  A-1 3.  Coefficient  Alphas  of  the  Three-Item  Quiz,  by  Part  B  Utilization1 


Interview  Type/Survey  Year 

No  Utilization 

Some  Utilization 

Sample  Member  Interviews 

1998  (Round  24) 

0.52 

0.45 

1999  (Round  27) 

0.53 

0.44 

Proxy  Interviews 

1998  (Round  24) 

0.53 

0.50 

1999  (Round  27) 

0.52 

0.50 

1  Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
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Table  A-14.  Coefficient  Alphas  of  the  Three-Item  Quiz,  by  Total  Covered 
Charges1 


Interview  Type/Survey  Year 

$0 

$1  -  $499 

$500  -  $4,999 

$5,000  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.52 

0.47 

0.43 

0.47 

1999  (Round  27) 

0.56 

0.38 

0.46 

0.44 

Proxy  Interviews 

1998  (Round  24) 

0.54 

0.52 

0.53 

0.43 

1999  (Round  27) 

0.51 

0.61 

0.52 

0.38 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 


Table  A-1 5.  Coefficient  Alphas  of  the  Three-Item  Quiz,  by  Total  Reimbursed 
Dollars1 


Interview  Type/Survey  Year 

$0 

$1  -  $499 

$500  -  $4,999 

$5,000  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.51 

0.45 

0.44 

0.46 

1999  (Round  27) 

0.52 

0.43 

0.44 

0.43 

Proxy  Interviews 

1998  (Round  24) 

0.51 

0.56 

0.47 

0.48 

1999  (Round  27) 

0.46 

0.60 

0.53 

0.32 

1  Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 


Table  A-1 6.   Coefficient  Alphas  of  the  Three-Item  Quiz,  by  Covered 
Institutional  Charges1 


Interview  Type/Survey  Year 

$0 

$1  -  $499 

$500  -  $4,999 

$5,000  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.50 

0.43 

0.46 

0.47 

1999  (Round  27) 

0.48 

0.40 

0.46 

0.45 

Proxy  Interviews 

1998  (Round  24) 

0.56 

0.47 

0.48 

0.47 

1999  (Round  27) 

0.54 

0.58 

0.56 

0.36 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
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Table  A-17.   Coefficient  Alphas  of  the  Three-Item  Quiz,  by  Allowed  Part  B 
Charges1 


Interview  Type/Survey  Year 

$0 

$1  - $499 

$500 -$1,499 

$1,500  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.53 

0.46 

0.41 

0.46 

1999  (Round  27) 

0.55 

0.43 

0.43 

0.44 

Proxy  Interviews 

1998  (Round  24) 

0.51 

0.53 

0.47 

0.48 

1999  (Round  27) 

0.53 

0.54 

0.56 

0.36 

1 0nly  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
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Table  A-18.  Results  from  Statistical  Analysis  of  Three-Item  Quiz  among 
Sample  Members 


Variable 

Chi-Square  Value 

Degrees  of 
Freedom 

P-Value 

Sample  Member  Interviews  - 1998 

Education 

575.82 

6 

.001 

Age  category 

93.09 

3 

<.001 

Income  category 

418.82 

3 

.001 

Managed  care 

37.94 

3 

.001 

Private  supplemental  insurance 

214.15 

3 

<.001 

Any  institutional  utilization 

110.31 

3 

.001 

Any  Part  B  claim 

355.89 

3 

.001 

Total  covered  charges 

.381.88 

9 

.001 

Total  reimbursed  dollars 

391.32 

9 

.001 

Covered  institutional  charges1 

118.90 

9 

.001 

Allowed  Part  B  charges 

402.02 

9 

.001 

Sample  Member  Interviews  - 1999 

Education 

179.65 

6 

<.001 

Age  category 

31.44 

3 

<  .001 

Income  category 

159.57 

3 

<  .001 

Managed  care 

6.06 

3 

n.s. 

Private  supplemental  insurance 

19.04 

3 

<.001 

Any  institutional  utilization1 

43.97 

3 

<.001 

Any  Part  B  claim 

66.41 

3 

<  .001 

1 

Total  covered  charges 

94.23 

9 

<.001 

Total  reimbursed  dollars 

102.69 

9 

<.001 

Covered  institutional  charges' 

46.42 

9 

<.001 

Allowed  Part  B  charges1 

88.13 

9 

<.001 

1  Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
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Table  A-19.  Results  from  Statistical  Analysis  of  Three-Item  Quiz  among 
Proxy  Respondents 


Variable 

Chi-Square  Value 

Degrees  of 
Freedom 

P-Value 

Proxy  Interviews  - 1998 

Education 

37.21 

6 

.001 

Age  category 

3.51 

3 

n.s. 

Income  category 

67.49 

3 

.001 

Managed  care 

2.89 

3 

n.s. 

Private  supplemental  insurance 

37.63 

3 

<.001 

Any  institutional  utilization 

15.27 

3 

.002 

Any  Part  B  claim 

21.98 

3 

.001 

Total  covered  charges 

32.03 

9 

.001 

Total  reimbursed  dollars 

32.08 

9 

.001 

Covered  institutional  charges 

26.01 

9 

.002 

Allowed  Part  B  charges 

34.14 

9 

.001 

Proxy  Interviews  - 1999 

Education 

12.30 

6 

.056 

Age  category 

3.98 

3 

n.s. 

Income  category 

19.93 

3 

<.001 

Managed  care 

5.46 

3 

n.s. 

Private  supplemental  insurance 

11.74 

3 

.008 

Any  institutional  utilization 

6.28 

3 

n.s. 

Any  Part  B  claim 

12.15 

3 

.007 

Total  covered  charges 

18.59 

9 

.029 

Total  reimbursed  dollars 

22.32 

9 

.008 

Covered  institutional  charges 

10.41 

9 

n.s. 

Allowed  Part  B  charges 

23.47 

9 

.005 

1  Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
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Table  A-20.   Coefficient  Alphas  of  the  Seven-Item  Quiz,  by  Enrollment  in 
Managed  Care  During  the  Past  Year 


Interview  Type/Survey  Year 

No  Enrollment 

Some  Enrollment 

Sample  Member  Interviews 

1998  (Round  24) 

0.74 

0.69 

1999  (Round  27) 

0.72 

0.70 

Proxy  Interviews 

1998  (Round  24) 

0.77 

0.76 

1999  (Round  27) 

0.75 

0.71 

Table  A-21.  Coefficient  Alphas  of  the  Seven-Item  Quiz,  by  Institutional 
Utilization1 


Interview  Type/Survey  Year 

No  Utilization 

Some  Utilization 

Sample  Member  Interviews 

1998  (Round  24) 

0.75 

0.74 

1999  (Round  27) 

0.74 

0.71 

Proxy  Interviews 

1998  (Round  24) 

0.77 

0.76 

1999  (Round  27) 

0.75 

0.75 

1  Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 


Table  A-22.  Coefficient  Alphas  of  the  Seven-Item  Quiz,  by  Part  B 
Utilization1 


Interview  Type/Survey  Year 

No  Utilization 

Some  Utilization 

Sample  Member  Interviews 

1998  (Round  24) 

0.78 

0.73 

1999  (Round  27) 

0.76 

0.72 

Proxy  Interviews 

1998  (Round  24) 

0.79 

0.76 

1999  (Round  27) 

0.75 

0.75 

1  Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
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Table  A-23.   Coefficient  Alphas  of  the  Seven-Item  Quiz,  by  Total  Covered 
Charges1 


Interview  Type/Survey  Year 

$0 

$1  - $499 

$500  -  $4,999 

$5,000  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.78 

0.73 

0.73 

0.74 

1999  (Round  27) 

0.77 

0.71 

0.72 

0.72 

Proxy  Interviews 

1998  (Round  24) 

0.78 

0.73 

0.77 

0.77 

1999  (Round  27) 

0.76 

0.73 

0.74 

0.77 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 


Table  A-24.  Coefficient  Alphas  of  the  Seven-Item  Quiz,  by  Total 
Reimbursed  Dollars1 


Interview  Type/Survey  Year 

$0 

$1  -  $499 

$500  -  $4,999 

$5,000  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.78 

0.73 

0.73 

0.75 

1999  (Round  27) 

0.75 

0.71 

0.71 

0.73 

Proxy  Interviews 

1998  (Round  24) 

0.78 

0.74 

0.78 

0.77 

1999  (Round  27) 

0.74 

0.74 

0.75 

0.78 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 


Table  A-25.   Coefficient  Alphas  of  the  Seven-Item  Quiz,  by  Covered 
Institutional  Charges1 


Interview  Type/Survey  Year 

$0 

$1  - $499 

$500  -  $4,999 

$5,000  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.75 

0.73 

0.73 

0.74 

1999  (Round  27) 

0.74 

0.71 

0.70 

0.72 

Proxy  Interviews 

1998  (Round  24) 

0.77 

0.75 

0.76 

0.77 

1999  (Round  27) 

0.75 

0.76 

0.74 

0.75 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
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Table  A-26.  Coefficient  Alphas  of  the  Seven-Item  Quiz,  by  Allowed  Part  B 
Charges1 


Interview  Type/Survey  Year 

$0 

$1  -  $499 

$500 -$1,499 

$1,500  or  More 

Sample  Member  Interviews 

1998  (Round  24) 

0.78 

0.73 

0.73 

0.74 

1999  (Round  27) 

0.76 

0.71 

0.71 

0.73 

Proxy  Interviews 

1998  (Round  24) 

0.78 

0.74 

0.77 

0.77 

1999  (Round  27) 

0.77 

0.74 

0.73 

0.76 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these  analyses. 
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Table  A-27,  Mean  Seven-Item  Quiz  Scores  and  Statistical  Significance  of 
ANOVAS  and  t-tests  among  Sample  Members 


Variable 

Sample  Members 
1998 

Sample  Members 
1999 

Education 

.v  -'■'  ','..-          .-                                          :■    ■ 

8th  grade  or  less 

2.86*** 

2.67*** 

More  than  8th  grade,  but  no  college 

3.76 

3.75 

College 

4.34 

4.26 

Age  category 

65-75 

4.10*** 

4.10*** 

Over  75 

3.58 

3.50 

Income  category 

Under  $25,000 

3.48*** 

3  42*** 

$25,000  or  more 

4.40 

4.32 

Managed  care 

■'■■'"■'       "                 ■■              ■■  ■       ■   '         ■       " .      .    '■■ 

Some  enrollment 

4.67*** 

4.61*** 

No  enrollment 

3.55 

3.52 

Private  Supplemental  Insurance 

Enrolled 

4.16*** 

4  i i *** 

Not  enrolled 

3.47 

3.37 

Any  institutional  utilization  \ 

Some  utilization 

3.54 

3.54 

No  utilization 

3.58 

3.47 

Any  Part  B  utilization  * 

Some  utilization 

3.61*** 

3.54 

No  utilization 

3.13 

3.35 

Total  covered  charges ' 

$0 

T     1Q*** 

3.26* 

$l-$499 

3.54 

3.56 

$500-$4,999 

3.70 

3.62 

$5,000  or  more 

3.53 

3.43 

Covered  institutional  charges ' 

$0 

3.58** 

3.48 

$l-$499 

3.64 

3.67 

$500-$4,999 

3.55 

3.53 

$5,000  or  more 

3.43 

3.42 



Allowed  Part  B  charges  ! 

$0 

"J     IT*** 

3.29* 

$l-$499 

3.51 

3.49 

$500-$  1,499 

3.72 

3.65 

$1,500  or  more 

3.60 

3.52 

Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in 
these  analyses. 
*  p  <  .05,  **  p  <  .01,  ***  p  <  .001 
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Table  A-28.   Mean  Seven-Item  Quiz  Scores  and  Statistical  Significance  of 
ANOVAS  and  t-tests  among  Proxy  Respondents 


Variable 

Proxy  Respondents 
1998 

Proxy  Respondents 
1999 

Education 

8th  grade  or  less 

3.08*** 

3.20** 

More  than  8th  grade,  but  no  college 

3.84 

3.63 

College 

4.11 

4.34 

Age  category 

• 

65-75 

3.64 

4.10 

Over  75 

3.64 

3.51 

Income  category 

Under  $25,000 

3.33*** 

2  29*** 

$25,000  or  more 

4.48 

4.64 

Managed  care 

Some  enrollment 

4.38*** 

4  52*** 

No  enrollment 

3.44 

3.36 

Private  Supplemental  Insurance 

■ 

Enrolled 

4.20*** 

4.39** 

Not  enrolled 

3.27 

3.40 

Any  institutional  utilization  ' 

Some  utilization 

3.36 

3.40 

No  utilization 

3.49 

3.30 

Any  Part  B  utilization  1 

Some  utilization 

3.52** 

3.44 

No  utilization 

3.03 

2.95 

Total  covered  charges  ! 

■.■.',■         ■         '     ■.'     ;     "■■--.' 

$0 

3.13 

3.00 

$l-$499 

3.28 

3.17 

$500-$4,999 

3.58 

3.37 

$5,000  or  more 

3.53 

3.64 

Covered  institutional  charges ' 

$0 

3.37 

3.30 

$l-$499 

3.29 

3.17 

$500-$4,999 

3.56 

3.25 

$5,000  or  more 

3.54 

3.66 

Allowed  Part  B  charges  ' 

$0 

3.05** 

2.86 

$l-$499 

3.34 

3.33 

$500-$  1,499 

3.71 

3.48 

$1,500  or  more 

3.53 

3.57 

1  Only  respondents  who  were  not  enrolled  in  managed  care  during  the  past  year  were  included  in  these 

analyses. 
*  p  <  .05,  **  p  <  .01,  ***  p  <  .001 
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